Enrichment Latency Budgets for Real-Time Correlation

Real-time correlation lives or dies on a latency budget, and threat-intel enrichment is the step most likely to blow it — a single slow feed lookup can add hundreds of milliseconds to every event. This guide builds an enrichment path with an explicit latency budget, as one precise technique within threat intel feed mapping and the broader SOC log architecture & taxonomy discipline.

Root-Cause Context

Correlation that must fire within, say, 500 ms of an event cannot afford to spend 300 ms of that budget waiting on a threat-intel API. Yet the naive enrichment path does exactly that: it calls the feed synchronously per event, with no deadline, so p99 latency is hostage to the slowest external dependency. Under an incident storm — precisely when correlation matters most — the feed’s own latency climbs, and enrichment stalls the entire correlation loop behind it.

The fix is to treat latency as a budget that is spent and accounted for. Enrichment is served from a tiered cache — an in-process LRU first, a shared Redis cache second, the authoritative feed last — and each tier has a deadline, with the total bounded so enrichment can never consume more than its allotted slice. Crucially, the path is fail-open: if the budget is exhausted before the feed answers, correlation proceeds on whatever was cached, because a decision made 100 ms late on full context is worse than one made on time with partial context.

Prerequisites

The pattern targets Python 3.11+ with async caching. It uses redis for the shared tier and the standard library elsewhere, and plugs into the parent threat intel feed mapping guides.

pip install "redis>=5.0,<6.0"

Production-Ready Implementation

The lookup walks the tiers under a shared deadline, returning the best answer available when the budget is spent and back-filling the faster tiers asynchronously after a feed hit.

from __future__ import annotations

import asyncio
import logging
import time
from collections import OrderedDict
from typing import Awaitable, Callable, Optional

logger = logging.getLogger("soc.intel.latency")


class LruCache:
    def __init__(self, capacity: int = 100_000) -> None:
        self._d: OrderedDict[str, dict] = OrderedDict()
        self._cap = capacity

    def get(self, key: str) -> Optional[dict]:
        if key not in self._d:
            return None
        self._d.move_to_end(key)
        return self._d[key]

    def put(self, key: str, value: dict) -> None:
        self._d[key] = value
        self._d.move_to_end(key)
        if len(self._d) > self._cap:
            self._d.popitem(last=False)


class BudgetedEnricher:
    def __init__(
        self, l1: LruCache,
        l2_get: Callable[[str], Awaitable[Optional[dict]]],
        feed_get: Callable[[str], Awaitable[Optional[dict]]],
        budget_ms: float = 50.0,
    ) -> None:
        self._l1 = l1
        self._l2_get = l2_get
        self._feed_get = feed_get
        self._budget = budget_ms / 1000.0

    async def enrich(self, indicator: str) -> tuple[Optional[dict], str]:
        """Return (verdict, tier). Fail open when the budget is exhausted."""
        start = time.perf_counter()

        hit = self._l1.get(indicator)
        if hit is not None:
            return hit, "L1"

        remaining = self._budget - (time.perf_counter() - start)
        try:
            hit = await asyncio.wait_for(self._l2_get(indicator), max(remaining, 0.001))
        except asyncio.TimeoutError:
            hit = None
        if hit is not None:
            self._l1.put(indicator, hit)
            return hit, "L2"

        remaining = self._budget - (time.perf_counter() - start)
        if remaining <= 0:
            logger.info("ERR_INTEL_411 budget exhausted indicator=%s", indicator)
            return None, "budget_exhausted"        # fail open, no verdict
        try:
            verdict = await asyncio.wait_for(self._feed_get(indicator), remaining)
        except asyncio.TimeoutError:
            logger.info("ERR_INTEL_412 feed deadline indicator=%s", indicator)
            return None, "feed_timeout"            # fail open
        if verdict is not None:
            self._l1.put(indicator, verdict)       # back-fill fast tier
        return verdict, "L3"


async def _demo() -> None:
    l1 = LruCache()
    async def l2(_k: str) -> Optional[dict]:
        return None
    async def feed(_k: str) -> Optional[dict]:
        await asyncio.sleep(0.01)
        return {"verdict": "malicious"}
    enricher = BudgetedEnricher(l1, l2, feed, budget_ms=50)
    verdict, tier = await enricher.enrich("203.0.113.9")
    assert verdict["verdict"] == "malicious" and tier == "L3"
    verdict2, tier2 = await enricher.enrich("203.0.113.9")
    assert tier2 == "L1"                            # back-filled, now instant


if __name__ == "__main__":
    asyncio.run(_demo())

The budget is measured once and decremented across tiers, so the feed lookup only gets whatever time remains after the caches — it can never blow the total. A feed hit back-fills L1 immediately, so the second lookup of a hot indicator is sub-millisecond, which is what keeps steady-state enrichment far under budget.

Error-Code Reference

Operational Notes

• Spend the budget, do not exceed it. Decrement one total budget across tiers so the slow feed only ever gets the time left after the caches.
• Fail open on exhaustion. A correlation decision made on time with partial context beats a late decision with full context; never let enrichment stall the loop.
• Back-fill the fast tiers asynchronously. After a feed hit, populate L1 and L2 so the next lookup of that indicator is instant — this is what keeps steady-state latency low.
• Circuit-break a slow feed. When the feed’s latency or error rate crosses a threshold, open a breaker and serve cache-only, so one degraded feed cannot drag every event’s enrichment.

Verification Checklist

• A cold indicator resolves via the feed within the total budget.
• The same indicator resolves from L1 on the next lookup (back-fill works).
• Budget exhaustion returns no verdict and fails open (ERR_INTEL_411), never blocks.
• A slow feed hits its remaining-time deadline and fails open (ERR_INTEL_412).
• An L2 outage falls through to the feed without exceeding the budget.
• Steady-state p99 enrichment latency stays well under the budget.

FAQ

Related

• Threat Intel Feed Mapping
• Mapping MISP Feeds to ECS
• Case Enrichment Pipelines
• Async Log Batching
• SOC Log Architecture & Taxonomy