Introduction
The Elastic Common Schema (ECS) provides an open, consistent model for structuring your data in the Elastic Stack. By normalizing data to a single common model, you can uniformly examine your data using interactive search, visualizations, and automated analysis.
Elastic provides hundreds of integrations that are ECS-compliant out-of-the-box, but ECS also allows you to normalize custom data sources. Normalizing a custom source can be an iterative and sometimes time-intensive process. However, we can use the Elastic Security Detection Engine to help quickly identify ECS non-compliance in our events.
First Things
First, let’s define ECS compliance. “ECS-compliant” describes indexed data that adopts and aligns with the ECS requirements and guidelines. ECS-compliant events ensure the best possible experience from your data across the Elastic Stack and solutions.
If you’d like to experiment with any of the detection rules demonstrated here, the easiest way to get started is by signing up for an Elastic Cloud free trial. All of the sample detection rules covered here can be downloaded from this gist and imported into the Detection Engine.
Now, let’s explore a few examples of detection rules which will alert on ECS non-compliant events.
Missing Fields
An ECS event should populate the ecs.version field. By having each event include the targeted ECS version, we know and understand what fields and data types to expect or not expect.
In this first example, detection alerts will be generated if the ecs.version field is missing from any event in the targeted indices:
not ecs.version:*
Shortly after activating the rule, alerts are generated for non-compliant events. With the details from the alerts, we can know exactly which data sources and events are non-compliant:
Beyond a single missing field, detections can also identify the absence of expected fields when others are present. For example, this next detection looks for an event that populates event.category: process but fails to populate some key process.* fields:
event.category: process and not (process.args: * and process.executable: * and process.name: * and process.pid: * and process.title: * and process.working_directory: *)
Taking a look at the generated alerts, we discover our source isn’t populating the process.pid field:
Allowed Values
Data sources can populate the ECS categorization fields to describe and better classify each event. Populating each event with the proper fields and values ensures consistency to group related event types.
This next rule checks each event containing the event.category field is using a valid allowed value:
event.category:* and not event.category:authentication and not event.category:configuration and not event.category:database and not event.category:driver and not event.category:file and not event.category:host and not event.category:iam and not event.category:intrusion_detection and not event.category:malware and not event.category:network and not event.category:package and not event.category:process and not event.category:web
Events with an invalid value for the event.category field are now generating alerts. In our example, event.category is set to use “auth” instead of the correct value, “authentication.” Now armed with this insight, we can turn attention to correcting the problem instead of hunting for non-compliant events.
Source and Destination Pairings
Not all events are network events, but they should always populate the source and destination as a pair when they are. Given that our custom event source contains network events, here's a rule that ensures source and destination are both populated:
(source.address: * and not destination.address: * ) or (destination.address: * and not source.address:*)
After activating the new detection, the new detection alerts indicate the source.address field is not populated:
Breakdown Fields
Some ECS fields can be populated by an ingest pipeline processor, like user agent. When the original User-Agent field runs through the user-agent ingest processor, the processor will parse then populate data into the appropriate fields.
This last example looks for the expected field breakdown and will alert if an expected field is missing in an event:
user_agent.original: * and not (user_agent.device.name: * and user_agent.name: * and user_agent.original: * and user_agent.os.full: * and user_agent.os.name: * and user_agent.os.version: * and user_agent.version: *)
Alerts are created after the rule is activated. Digging into the alert, we confirm that the event is missing the user_agent.name field:
Summary
These few examples are just starting points. With custom ECS compliance-focused detections, the Detection Engine becomes another tool to improve and refine your data ingestion process continually. Just make sure these ECS detection rules don’t trigger a pager alert!