# Dec 4th, 2020: \[EN\] Validate Elastic Common Schema (ECS) fields using Security Detection Rules

**URL:** <https://discuss.elastic.co/t/dec-4th-2020-en-validate-elastic-common-schema-ecs-fields-using-security-detection-rules/254805>\
**Category:** Advent Calendar\
**Tags:** ecs-elastic-common-schema\
**Created:** [December 4, 2020, 8:00am UTC](https://discuss.elastic.co/t/dec-4th-2020-en-validate-elastic-common-schema-ecs-fields-using-security-detection-rules/254805 "2020-12-04T08:00:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ebeahan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebeahan/32/78989_2.png) [@ebeahan](https://discuss.elastic.co/u/ebeahan)\
**Post date:** [December 4, 2020, 8:00am UTC](https://discuss.elastic.co/t/dec-4th-2020-en-validate-elastic-common-schema-ecs-fields-using-security-detection-rules/254805/1 "2020-12-04T08:00:00Z")

</div>

**Introduction**

The Elastic Common Schema (ECS) provides an open, consistent model for structuring your data in the Elastic Stack. By normalizing data to a single common model, you can uniformly examine your data using interactive search, visualizations, and automated analysis.

Elastic provides hundreds of [integrations](https://www.elastic.co/integrations) that are ECS-compliant out-of-the-box, but ECS also allows you to normalize custom data sources. Normalizing a custom source can be an iterative and sometimes time-intensive process. However, we can use the Elastic Security Detection Engine to help quickly identify ECS non-compliance in our events.

**First Things**

First, let’s define ECS compliance. “ECS-compliant” describes indexed data that adopts and aligns with the [ECS requirements and guidelines](https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html#_general_guidelines). ECS-compliant events ensure the best possible experience from your data across the Elastic Stack and solutions.

If you’d like to experiment with any of the detection rules demonstrated here, the easiest way to get started is by [signing up](https://www.elastic.co/cloud/elasticsearch-service/signup) for an Elastic Cloud free trial. All of the sample detection rules covered here can be downloaded from [this gist](https://gist.github.com/ebeahan/fdbee3a5d903b55d7e401046ea23d382) and [imported](https://www.elastic.co/guide/en/security/current/rules-ui-management.html#import-export-rules-ui) into the Detection Engine.

Now, let’s explore a few examples of detection rules which will alert on ECS non-compliant events.

**Missing Fields**

An ECS event should populate the `ecs.version` field. By having each event include the targeted ECS version, we know and understand what fields and data types to expect or not expect.

In this first example, detection alerts will be generated if the `ecs.version` field is missing from any event in the targeted indices:

```auto
not ecs.version:*

```

 ![missing-ecs-version-rule](https://us1.discourse-cdn.com/elastic/original/3X/e/6/e69ef31b06078bae4abaab6601735f07b80b8e0d.png)

Shortly after activating the rule, alerts are generated for non-compliant events. With the details from the alerts, we can know exactly which data sources and events are non-compliant:

 ![missing-ecs-version-alerts](https://us1.discourse-cdn.com/elastic/original/3X/4/2/426f474851fa09ad8a7e43aafea2517be9a7641f.png)

Beyond a single missing field, detections can also identify the absence of expected fields when others are present. For example, this next detection looks for an event that populates `event.category: process` but fails to populate some key `process.*` fields:

```auto
event.category: process and not (process.args: * and process.executable: * and process.name: * and process.pid: * and process.title: * and process.working_directory: *)

```

 ![process-rule](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af083295413f4a489c1e6f93def8dc5a8f7dad2b.png)

Taking a look at the generated alerts, we discover our source isn’t populating the `process.pid` field:

 ![process-alerts](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bbae50acf57924dd51fee0daa05b4d74299a086d.png)

**Allowed Values**

Data sources can populate the [ECS categorization fields](https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html) to describe and better classify each event. Populating each event with the proper fields and values ensures consistency to group related event types.

This next rule checks each event containing the `event.category` field is using a valid [allowed value](https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-category.html):

```auto
event.category:* and not event.category:authentication and not event.category:configuration and not event.category:database and not event.category:driver and not event.category:file and not event.category:host and not event.category:iam and not event.category:intrusion_detection and not event.category:malware and not event.category:network and not event.category:package and not event.category:process and not event.category:web

```

 ![ecs-invalid-event-category-rule](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f7669f0b9e3c22a5fbb81647332cc23dd92a278.png)

Events with an invalid value for the `event.category` field are now generating alerts. In our example, `event.category` is set to use “auth” instead of the correct value, “authentication.” Now armed with this insight, we can turn attention to correcting the problem instead of hunting for non-compliant events.

 ![ecs-invalid-event-category-alerts](https://us1.discourse-cdn.com/elastic/original/3X/8/d/8d26587ef86614dbb31ec407873c45b97aaa9da6.png)

**Source and Destination Pairings**

Not all events are network events, but they should always populate the source and destination as a pair when they are. Given that our custom event source contains network events, here's a rule that ensures source and destination are both populated:

```auto
(source.address: * and not destination.address: * ) or (destination.address: * and not source.address:*)

```

 ![source-dest-rule](https://us1.discourse-cdn.com/elastic/original/3X/2/c/2c22b18807abb82fca57f5a592d6a90aeae05484.png)

After activating the new detection, the new detection alerts indicate the `source.address` field is not populated:

 ![ecs-source-dest-alerts](https://us1.discourse-cdn.com/elastic/original/3X/6/9/69c363756755ab37361e60b3bf47a78df0f86008.png)

**Breakdown Fields**

Some ECS fields can be populated by an ingest pipeline processor, like [user agent](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/user-agent-processor.html). When the original User-Agent field runs through the user-agent ingest processor, the processor will parse then populate data into the appropriate fields.

This last example looks for the expected field breakdown and will alert if an expected field is missing in an event:

```auto
user_agent.original: * and not (user_agent.device.name: * and user_agent.name: * and user_agent.original: * and user_agent.os.full: * and user_agent.os.name: * and user_agent.os.version: * and user_agent.version: *)

```

 ![user-agent-rule](https://us1.discourse-cdn.com/elastic/original/3X/5/c/5c80965b619487d50ea2562a6c45ccb8b3e5a945.png)

Alerts are created after the rule is activated. Digging into the alert, we confirm that the event is missing the `user_agent.name` field:

 ![user-agent-alerts](https://us1.discourse-cdn.com/elastic/original/3X/6/7/678b71795954e0f183c82822416dc75771e8d523.png)

 ![user-agent-fields](https://us1.discourse-cdn.com/elastic/original/3X/4/e/4ea38c7992ce8d1e52e5edc69d12a836cc49043e.png)

**Summary**

These few examples are just starting points. With custom ECS compliance-focused detections, the Detection Engine becomes another tool to improve and refine your data ingestion process continually. Just make sure these ECS detection rules don’t trigger a pager alert!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-4th-2020-en-validate-elastic-common-schema-ecs-fields-using-security-detection-rules/254805/2 "2021-01-01T08:00:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.