# Elastic enrich data based on two matching fields

**URL:** <https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [February 15, 2023, 8:41am UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561 "2023-02-15T08:41:05Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![maggo](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@maggo](https://discuss.elastic.co/u/maggo)\
**Post date:** [February 15, 2023, 8:41am UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/1 "2023-02-15T08:41:05Z")

</div>

Hello guys,

i'm pretty new to ELK and want to implement a vulnerability enrichment for incoming osquery data.

I have one index with vulnerability data with fields like:

```auto
"affected_product": "chrome"
"affected_version": "1.5"
"CVEID": "CVE-XXXX-XXXX"

```

Now i push data into a new index (osquery) with data like and use a pipeline to enrich:

```auto
"product_name": "chrome"
"version": 1.5"

```

With an enrich processor and a enrich policy it is possible to enrich the osquery data with the cveid base on the product\_name/affected\_product matching field. That is no problem.

But i need a second matching field. It should ONLY enrich data if BOTH fields (name & version) are matching.  
That means the enrich processor should check if the product\_name matches the affected\_product AND the version matches the affected\_version.  
If i set an "if" condition to the enrich processor it is possible to check the version field. But only for static/hardcoded values.

I tried a few things but could not get it running properly.

Feel free to ask for further information. I hope you guys can understand what im trying to do.

Kind regards,  
Marco

```auto
# Create enrichment policy
PUT /_enrich/policy/test-oval-cve-debian7-policy
{
	"match": {
		"indices": "test_oval_cve_debian7",
		"match_field": "affected_product",
		"enrich_fields": ["cveid","cveurl","debian_moreinfo"]
	}
}

# Create ingestor pipeline with enrich processor
PUT /_ingest/pipeline/test-oval-cve-debian7-pipeline
{
	"processors" : [
		{
			"enrich" : {
				"description": "Add cve data based on product",
				"policy_name": "test-oval-cve-debian7-policy",
				"field" : "product_name",
				"target_field":"cve",
				"max_matches": 1
			}
		}
		  
	]
}

#working with static second matching value
PUT /_ingest/pipeline/test-oval-cve-debian7-version-lookup
{
	"processors" : [
		{
			"pipeline" : {
				"description": "Execute pipeline based on matching version",
				"if": "ctx.product_version == '1:1.2.11.dfsg-2+deb11u2'",
				"name": "test-oval-cve-debian7-pipeline"
			}
		}
	]
}

# Add document to index test_osquery with static version check lookup
PUT /test_osquery/_doc/7?pipeline=test-oval-cve-debian7-version-lookup
{
	"product_name": "zlib1g",
	"product_version": "1:1.2.11.dfsg-2+deb11u2"
}

```

---

<div class="post-metadata">

**Author:** ![Sebastian\_Huettersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_huettersen/32/117296_2.png) [@Sebastian\_Huettersen](https://discuss.elastic.co/u/Sebastian_Huettersen)\
**Post date:** [February 24, 2023, 12:00pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/2 "2023-02-24T12:00:43Z")

</div>

I believe that is not currently possible see:

> <https://github.com/elastic/elasticsearch/issues/48988>
>
> The issue contains improvements and new features that should be done for ingest …in near to medium future.
> 
> \- \[\] Add telemetry support for the enrich policy. Interesting information to capture is the number of enrich policies by policy type, last policy execution by type and size (doc count & index size in bytes) of the enrich policy indices per type. 
> \- \[x\] Add range enrich policy type. The range policy allows for example to match whether an ip falls into a certain range. The match field can be of type: \`ip\_range\`, \`date\_range\`, \`float\_range\`, \`long\_range\`, \`integer\_range\` and \`double\_range\`. #76110
> \- \[\] Extend the get policies api to include the status of a policy and the last time a policy executed.
> \- \[\] Optimize the enrich processor queue, so that when multiple search requests are executed then to combine identical search requests into a single search request in order to avoid duplicate searches in the same multi search request.
> \- \[x\] Introduce a LRU cache to avoid repeatable searches from the enrich processor. #76800
> \- \[x\] Allow enrich policy execution to be cancellable. #77188
> \- \[\] Store the result of a policy execution in the .tasks index.
> \- \[\] Explore allowing multiple match fields and a combination of policy types. To allow more dynamic querying. This could perhaps be exposed as composite policy type.
> \- \[\] Explore allowing enrich processor to query source index directly. In cases where flexibility is more important than performance. #73407
> \- \[\] Add more surface area for stats collection on enrich indices, including average search times, total search operations, and average size of search operations.

- Explore allowing multiple match fields and a combination of policy types. To allow more dynamic querying. This could perhaps be exposed as composite policy type.

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [February 25, 2023, 2:41pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/3 "2023-02-25T14:41:12Z")

</div>

This isn't directly possible, but a solution might be:

1. Create an Enrichment Policy whose `match_field` is `affected_version`.
  - An assumption here is that `affected_version` probably will produce the least number of duplicate matches for step 2

2. Add an enrich processor and set `max_matches` to a value greater than 1.
  - The exact number to set here is somewhat hard to determine, but you'd want it to be high enough to almost always guarantee you'll match the document which contains the desired `affected_product`. You could possibly just set it to the max of `128`, but I'm not sure _how_ this will impact performance.

3. Have a script processor loop through the output of the enrich processor and compare the values of `affected_product`, dropping any matches that don't equal.

You should now in theory have a document that is enriched by both `affected_version` and `max_matches`. This solution is nowhere near as efficient as a native multi-match enrich processor would be, but it at least allows for it to be doable.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 25, 2023, 4:45pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/4 "2023-02-25T16:45:48Z")

</div>

You could just concatenate both the fields into a new field in each index when you ingest both indices. Then the enrich would work fine.

Might even be able to do it with a mapped runtime in the lookup index... Hmmm perhaps I will test that later today

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 25, 2023, 5:35pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/5 "2023-02-25T17:35:51Z")

</div>

Oh and @Sebastian_Huettersen and @maggo Welcome to the community!!

---

<div class="post-metadata">

**Author:** ![Sebastian\_Huettersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_huettersen/32/117296_2.png) [@Sebastian\_Huettersen](https://discuss.elastic.co/u/Sebastian_Huettersen)\
**Post date:** [February 25, 2023, 7:06pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/6 "2023-02-25T19:06:26Z")

</div>

Here's my thought approach if anyone else wants to implement something similar.

With the oval data you quickly reach the limit of max match 128, if you don't change anything directly on the oval data. Therefore, I would recommend merging the oval IDs to:

`os_vendor`, `os_version`,`product_name`, `fixed_version`.

This way you can reduce the number of oval documents.

Next you have to create a possibility to create a match between the osquery data and the oval data. Here I would use a set processor in the ingest pipeline that combines `host.os.codename` and `product name`.

Versions we can not match because all versions in a certain version range are affected not only 1 version. If we in the current state could combine different policy-type 's like match and range that shouldent work in the most cases since the version nr. cannot depict as a range.

Next, create an enrich process with `"max_matches" : 128` and as field we can use the new created field with the combined value of OS and product name for matching.

Now comes the trick of a script processor with painless.

```
"script": {
          "description": "Check if Version still affected",
          "lang": "painless",
          "source": """
              // Method to determined if package is affected
              int checkPackage(){
                //Do magic
              }

              if(some checks if fields exist and have correct datatype){
                for (element in ctx['enrich_taged_field']){
                  if (checkPackage() < 0){
                    //Copy info to target field
                  }
                }
              }
          """
}

```

At the end remove the target\_field from the enrich processor.

@stephenb would be nice there would be more examples in the documentation about enrich especialy templates to bypass the current limentation from enrich.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 25, 2023, 7:46pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/7 "2023-02-25T19:46:50Z")

</div>

> [@Sebastian\_Huettersen](#):
>
> @stephenb would be nice there would be more examples in the documentation about enrich especialy templates to bypass the current limentation from enrich.

Agree, Feel Free to Open an Enhancement Request against the Documents and perhaps provide some sample use cases...

I am seeing more of these with CVE databases perhaps a feature request around CVE Matching Against the Elasticsearch Repository and tag is Security.

I am not familiar in detail with the OVAL database so a few things are unclear to me but here is an

I do see the new JSON format like this so perhaps where the range would make sense?

```auto
                "version": "1.0.0",
                "status": "affected",
                "lessThan": "1.0.6",

```

Example based on the OP @maggo question he did not specify ranges so I am not sure if that applies or not.

Here is some simple code, I am sure it will not solve the entire issue but perhaps it will provide some other ideas.

It does some processing when loading the CVE data  
Then it uses similar processing before calling the enrich processor

```auto
DELETE discuss-cve-data

PUT /discuss-cve-data
{
  "mappings": {
    "properties": {
      "CVEID": {
        "type": "keyword"
      },
      "affected_product": {
        "type": "keyword"
      },
      "affected_version": {
        "type": "keyword"
      },
      "notes": {
        "type": "keyword"
      },
      "cve_product_version_unique_id": {
        "type": "keyword"
      }
    }
  }
}

PUT _ingest/pipeline/discuss-cve-prep
{
  "processors": [
    {
      "set": {
        "field": "cve_product_version_unique_id",
        "value": "{{affected_product}} - {{affected_version}}"
      }
    }
  ]
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "chrome",
  "affected_version": "1.5",
  "CVEID": "CVE-1234-5678",
  "notes" : "Bad Chrome Bug"
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "chrome",
  "affected_version": "1.5",
  "CVEID": "CVE-1234-3333",
  "notes" : "Another Bad Chrome Bug"
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "chrome",
  "affected_version": "1.6",
  "CVEID": "CVE-1234-9999",
  "notes" : "Worse Chrome Bug"
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "safari",
  "affected_version": "3.2",
  "CVEID": "CVE-1234-8888",
  "notes" : "Bad Safari Bug"
}

GET discuss-cve-data/_search
{
  "fields": ["*"]
}

DELETE /_enrich/policy/discuss-cve-data-policy

# Create enrichment policy
PUT /_enrich/policy/discuss-cve-data-policy
{
	"match": {
		"indices": "discuss-cve-data",
		"match_field": "cve_product_version_unique_id",
		"enrich_fields": ["cveid","notes"]
	}
}

POST /_enrich/policy/discuss-cve-data-policy/_execute

DELETE _ingest/pipeline/discuss-cve-lookup

PUT _ingest/pipeline/discuss-cve-lookup
{
  "processors": [
{
      "set": {
        "field": "cve_product_version_unique_id",
        "value": "{{product}} - {{version}}"
      }
    },
    {
      "enrich": {
        "policy_name": "discuss-cve-data-policy",
        "field": "cve_product_version_unique_id",
        "target_field": "cve_data",
        "max_matches": 128
      }
    }
  ]
}

DELETE discuss-product-logs

PUT /discuss-product-logs
{
  "mappings": {
    "properties": {
      "product": {
        "type": "keyword"
      },
      "version": {
        "type": "keyword"
      },
      "message": {
        "type": "keyword"
      },
      "cve_product_version_unique_id": {
        "type": "keyword"
      }
    }
  }
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "chrome",
  "version": "1.5",
  "message": "this is a log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "chrome",
  "version": "1.6",
  "message": "this is another log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "chrome",
  "version": "1.7",
  "message": "this is another log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "safari",
  "version": "3.2",
  "message": "this is another log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "safari",
  "version": "3.3",
  "message": "this is another log line"
}

GET discuss-product-logs/_search

```

Results

```auto
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "discuss-product-logs",
        "_id": "nDcaioYBQV01Dpx0-vsY",
        "_score": 1,
        "_source": {
          "cve_data": [
            {
              "cve_product_version_unique_id": "chrome - 1.5",
              "notes": "Bad Chrome Bug"
            },
            {
              "cve_product_version_unique_id": "chrome - 1.5",
              "notes": "Another Bad Chrome Bug"
            }
          ],
          "product": "chrome",
          "message": "this is a log line",
          "version": "1.5",
          "cve_product_version_unique_id": "chrome - 1.5"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "nTcaioYBQV01Dpx0-vs_",
        "_score": 1,
        "_source": {
          "cve_data": [
            {
              "cve_product_version_unique_id": "chrome - 1.6",
              "notes": "Worse Chrome Bug"
            }
          ],
          "product": "chrome",
          "message": "this is another log line",
          "version": "1.6",
          "cve_product_version_unique_id": "chrome - 1.6"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "njcaioYBQV01Dpx0-vtL",
        "_score": 1,
        "_source": {
          "product": "chrome",
          "message": "this is another log line",
          "version": "1.7",
          "cve_product_version_unique_id": "chrome - 1.7"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "nzcaioYBQV01Dpx0-vtX",
        "_score": 1,
        "_source": {
          "cve_data": [
            {
              "cve_product_version_unique_id": "safari - 3.2",
              "notes": "Bad Safari Bug"
            }
          ],
          "product": "safari",
          "message": "this is another log line",
          "version": "3.2",
          "cve_product_version_unique_id": "safari - 3.2"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "oDcaioYBQV01Dpx0-vtj",
        "_score": 1,
        "_source": {
          "product": "safari",
          "message": "this is another log line",
          "version": "3.3",
          "cve_product_version_unique_id": "safari - 3.3"
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![maggo](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@maggo](https://discuss.elastic.co/u/maggo)\
**Post date:** [February 27, 2023, 2:32pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/8 "2023-02-27T14:32:40Z")

</div>

Hello stephenb,  
thanks for your help!

I'm working together with sebastian on this implementation. We figured out a few things.  
We could find a working solution to enrich osquery data with the affected oval data.

We need more time for testing and will come back later.

Kind regards,  
Marco

---

<div class="post-metadata">

**Author:** ![maggo](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@maggo](https://discuss.elastic.co/u/maggo)\
**Post date:** [February 27, 2023, 2:35pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/9 "2023-02-27T14:35:42Z")

</div>

Hello BenB196,  
thanks for your help.

Yes that was one first try to get the 1:1 version match running. After that we recognized that older versions of the application are also affected. So we implemented a version comparison as painless script processor. We need more testing on the script right now.

Have a nice day!  
Marco

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2023, 2:35pm UTC](https://discuss.elastic.co/t/elastic-enrich-data-based-on-two-matching-fields/325561/10 "2023-03-27T14:35:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.