Schema Mapping Comparisons: Choosing and Reconciling ECS and OSSEM

The single most consequential taxonomy decision a SOC makes is which target schema every source normalizes into — and getting it wrong means either re-mapping every source later or living with detections that cannot correlate across tools. This page is part of the broader SOC log architecture & taxonomy discipline, and it compares the two schemas most teams weigh — the Elastic Common Schema (ECS) and the Open Source Security Events Metadata (OSSEM) — as a decision framework rather than a doctrine, then builds a reconciled mapping layer that lets a SOC standardize on one while borrowing the other’s strengths.

Problem Framing

Consider a concrete decision. A SOC runs detections in an Elastic-based SIEM, threat-hunting notebooks that follow OSSEM’s data dictionary, and a data lake that ingests both. If ingestion normalizes to ECS only, the OSSEM-based hunts must translate every field at query time; if it normalizes to OSSEM only, the SIEM’s ECS-native detection content breaks. Picking arbitrarily and hard-coding it into every parser means that when the second consumer’s needs surface, the team faces a re-mapping of dozens of sources. And the two schemas genuinely differ: ECS is broad, product-driven, and optimized for the Elastic stack; OSSEM is security-event-focused, vendor-neutral, and richer for adversary-behavior modeling. A field like a process command line, a logon type, or a network direction is named and typed differently in each.

The job of a schema-mapping decision is to choose a canonical target deliberately, then provide a reconciled mapping so both consumers are served from one normalized event — without maintaining two parallel normalization passes. That requires three things: an explicit comparison of the schemas against the SOC’s actual consumers, a mapping layer that translates source fields into the canonical schema and can project into the other on demand, and deterministic handling of the fields where the two schemas genuinely conflict. The rest of this page builds exactly that.

Prerequisites & Environment

The reference mapping layer targets Python 3.11+ and uses only the standard library plus pydantic for the canonical contract. The field maps themselves are version-controlled data.

pip install "pydantic>=2.6,<3.0"

The one assumption is a canonical decision, made once and recorded: this guide standardizes on ECS as the storage schema and projects into OSSEM for consumers that need it, but the mechanism is symmetric.

Architecture Overview

Mapping is a translation stage after parsing and before storage. Source fields are mapped into the canonical schema through a versioned field map; a projection function derives the alternate schema’s view on demand, so both consumers read from one normalized event.

Step-by-Step Implementation

Step 1 — Define the canonical contract and a versioned field map

The canonical event is ECS-typed; the field map translates each source’s dialect into it, and is data so a mapping change is reviewable.

from __future__ import annotations

from typing import Optional

from pydantic import BaseModel, Field


class CanonicalEvent(BaseModel):
    """ECS-aligned canonical event — the single stored shape."""

    timestamp: str = Field(min_length=1)          # ECS @timestamp
    source_ip: Optional[str] = None               # ECS source.ip
    user_name: Optional[str] = None               # ECS user.name
    process_command_line: Optional[str] = None    # ECS process.command_line
    event_action: Optional[str] = None            # ECS event.action
    network_direction: Optional[str] = None       # ECS network.direction


# Per-source dialect -> canonical ECS field map (version-controlled data).
WINDOWS_MAP: dict[str, str] = {
    "TimeCreated": "timestamp",
    "IpAddress": "source_ip",
    "TargetUserName": "user_name",
    "CommandLine": "process_command_line",
    "EventID": "event_action",
}

Step 2 — Map source fields into the canonical schema

Mapping is a deterministic rename-and-type pass. An unmapped source field is not silently dropped; it is recorded so the map’s coverage can be measured.

import logging

logger = logging.getLogger("soc.taxonomy.mapping")


def map_to_canonical(raw: dict, field_map: dict[str, str]) -> tuple[dict, list[str]]:
    """Return (canonical_dict, unmapped_keys)."""
    canonical: dict = {}
    unmapped: list[str] = []
    for src_key, value in raw.items():
        target = field_map.get(src_key)
        if target is None:
            unmapped.append(src_key)              # tracked, not dropped
            continue
        canonical[target] = value
    return canonical, unmapped

Step 3 — Project the canonical event into the alternate schema

A projection derives the OSSEM view from the stored ECS event, so hunts get their expected field names without a second normalization pass.

# Canonical (ECS) -> OSSEM field names for the projection.
ECS_TO_OSSEM: dict[str, str] = {
    "source_ip": "SrcIpAddr",
    "user_name": "SubjectUserName",
    "process_command_line": "CommandLine",
    "event_action": "EventID",
    "network_direction": "Direction",
}


def project_ossem(event: CanonicalEvent) -> dict:
    """Derive an OSSEM-named view of a canonical ECS event on demand."""
    ecs = event.model_dump(exclude_none=True)
    return {ECS_TO_OSSEM.get(k, k): v for k, v in ecs.items()}

Step 4 — Resolve genuine schema conflicts deterministically

Where the schemas encode the same concept differently (an enumeration, a direction, a type), a conflict policy makes the resolution explicit and testable rather than accidental.

def normalize_direction(value: str) -> Optional[str]:
    """Map vendor/OSSEM direction values onto the ECS enumeration."""
    mapping = {
        "inbound": "inbound", "in": "inbound", "ingress": "inbound",
        "outbound": "outbound", "out": "outbound", "egress": "outbound",
    }
    normalized = mapping.get(value.strip().lower())
    if normalized is None:
        logger.warning("ERR_MAP_203 unknown direction=%r", value)
    return normalized


if __name__ == "__main__":
    raw = {"TimeCreated": "2026-04-09T00:00:00Z", "TargetUserName": "jdoe",
           "IpAddress": "10.0.0.5", "CommandLine": "whoami", "EventID": "4688"}
    canonical, unmapped = map_to_canonical(raw, WINDOWS_MAP)
    event = CanonicalEvent(**canonical)
    ossem_view = project_ossem(event)
    assert ossem_view["SubjectUserName"] == "jdoe"

Schema & Validation Integration

The canonical contract is the same one the rest of the taxonomy enforces: mapping feeds directly into JSON event normalization and is validated by the schema validation pipelines, so a field that maps to a type the canonical schema rejects fails at the gate rather than corrupting storage. Because both consumers read from one stored ECS event and the OSSEM view is a projection, there is exactly one source of truth, and a mapping change is a single reviewed update to the version-controlled field map. The unmapped-field list from step 2 is the coverage metric: a source with many unmapped keys needs map work, tracked through the error categorization frameworks.

Error Handling & DLQ Routing

Mapping failures are stable and queryable. Codes follow the ERR_CATEGORY_NNN convention.

The cardinal rule is one canonical truth, everything else derived. Storing two parallel normalizations is the anti-pattern this design exists to prevent; the alternate schema is always a projection, so the two views can never drift apart. Unmapped required fields (ERR_MAP_201) fail closed to the DLQ rather than being dropped, because a missing field is a silent detection gap.

Performance Tuning

Mapping is cheap per event; the cost is in map maintenance and projection frequency. Tune in this order:

• Compile the field map once into a dict lookup; never re-parse the map config per event.
• Project lazily. Derive the OSSEM view only for consumers that request it, not on every stored event, so the write path pays only for the canonical schema.
• Cache projections for hot events if the same event is queried repeatedly in both views, but never persist the projection as a second stored copy.
• Batch map validation at config-load time — validate the entire field map against the canonical schema before swapping it in, so a bad map never reaches the hot path.
• Measure map coverage per source and prioritize map work by the unmapped-field rate, which correlates directly with lost detection fidelity.

Verification & Observability

Confirm correctness with round-trip tests and production coverage counters.

def test_ecs_ossem_roundtrip_preserves_values() -> None:
    raw = {"TargetUserName": "alice", "IpAddress": "10.1.1.1",
           "TimeCreated": "2026-04-09T00:00:00Z"}
    canonical, unmapped = map_to_canonical(raw, WINDOWS_MAP)
    event = CanonicalEvent(**canonical)
    ossem = project_ossem(event)
    assert ossem["SubjectUserName"] == "alice"     # ECS user.name -> OSSEM
    assert ossem["SrcIpAddr"] == "10.1.1.1"
    assert "TargetUserName" not in unmapped        # it was mapped


def test_unknown_direction_defaults_safely() -> None:
    assert normalize_direction("sideways") is None  # ERR_MAP_203, not a crash

Operationally, emit map_events_total{source,result}, map_coverage_ratio{source} (mapped over total fields), map_dlq_total{code}, and projection_requests_total{schema}. A dropping map_coverage_ratio on a source signals a vendor field change; a rising ERR_MAP_204 collision rate signals two sources fighting over one canonical field.

Troubleshooting

• Hunts and detections disagree about a field. Root cause: two parallel normalizations drifted apart. Fix: store one canonical schema and derive the other as a projection so they cannot diverge.
• A source’s fields keep landing unmapped. Root cause: the field map lags the vendor’s format. Fix: monitor map_coverage_ratio and extend the map when it drops.
• A network-direction filter misses events. Root cause: vendor direction values fall outside the canonical enum. Fix: extend the direction conflict policy and default unknowns to a counted bucket, not silently.
• Two sources overwrite each other’s field. Root cause: both map to one canonical field with no precedence. Fix: document a precedence rule and raise ERR_MAP_204 on collision.
• A projection returns fewer fields than expected. Root cause: the requested field was never stored in canonical. Fix: confirm the source populates it and that the field map covers it.

FAQ

Related

• SOC Log Architecture & Taxonomy
• Elastic ECS vs OSSEM Field Mapping
• JSON Event Normalization
• Threat Intel Feed Mapping
• Schema Validation Pipelines