Elastic enrich data based on two matching fields

maggo · February 15, 2023, 8:41am

Hello guys,

i'm pretty new to ELK and want to implement a vulnerability enrichment for incoming osquery data.

I have one index with vulnerability data with fields like:

"affected_product": "chrome"
"affected_version": "1.5"
"CVEID":  "CVE-XXXX-XXXX"

Now i push data into a new index (osquery) with data like and use a pipeline to enrich:

"product_name": "chrome"
"version": 1.5"

With an enrich processor and a enrich policy it is possible to enrich the osquery data with the cveid base on the product_name/affected_product matching field. That is no problem.

But i need a second matching field. It should ONLY enrich data if BOTH fields (name & version) are matching.
That means the enrich processor should check if the product_name matches the affected_product AND the version matches the affected_version.
If i set an "if" condition to the enrich processor it is possible to check the version field. But only for static/hardcoded values.

I tried a few things but could not get it running properly.

Feel free to ask for further information. I hope you guys can understand what im trying to do.

Kind regards,
Marco

# Create enrichment policy
PUT /_enrich/policy/test-oval-cve-debian7-policy
{
	"match": {
		"indices": "test_oval_cve_debian7",
		"match_field": "affected_product",
		"enrich_fields": ["cveid","cveurl","debian_moreinfo"]
	}
}

# Create ingestor pipeline with enrich processor
PUT /_ingest/pipeline/test-oval-cve-debian7-pipeline
{
	"processors" : [
		{
			"enrich" : {
				"description": "Add cve data based on product",
				"policy_name": "test-oval-cve-debian7-policy",
				"field" : "product_name",
				"target_field":"cve",
				"max_matches": 1
			}
		}
		  
	]
}

#working with static second matching value
PUT /_ingest/pipeline/test-oval-cve-debian7-version-lookup
{
	"processors" : [
		{
			"pipeline" : {
				"description": "Execute pipeline based on matching version",
				"if": "ctx.product_version == '1:1.2.11.dfsg-2+deb11u2'",
				"name": "test-oval-cve-debian7-pipeline"
			}
		}
	]
}

# Add document to index test_osquery with static version check lookup
PUT /test_osquery/_doc/7?pipeline=test-oval-cve-debian7-version-lookup
{
	"product_name": "zlib1g",
	"product_version": "1:1.2.11.dfsg-2+deb11u2"
}

Sebastian_Huettersen · February 24, 2023, 12:00pm

I believe that is not currently possible see:

github.com/elastic/elasticsearch

Enrich processor followup work

opened 01:32PM - 12 Nov 19 UTC

martijnvg

Meta :Data Management/Ingest Node Team:Data Management

The issue contains improvements and new features that should be done for ingest …in near to medium future. - [ ] Add telemetry support for the enrich policy. Interesting information to capture is the number of enrich policies by policy type, last policy execution by type and size (doc count & index size in bytes) of the enrich policy indices per type. - [x] Add range enrich policy type. The range policy allows for example to match whether an ip falls into a certain range. The match field can be of type: `ip_range`, `date_range`, `float_range`, `long_range`, `integer_range` and `double_range`. #76110 - [ ] Extend the get policies api to include the status of a policy and the last time a policy executed. - [ ] Optimize the enrich processor queue, so that when multiple search requests are executed then to combine identical search requests into a single search request in order to avoid duplicate searches in the same multi search request. - [x] Introduce a LRU cache to avoid repeatable searches from the enrich processor. #76800 - [x] Allow enrich policy execution to be cancellable. #77188 - [ ] Store the result of a policy execution in the .tasks index. - [ ] Explore allowing multiple match fields and a combination of policy types. To allow more dynamic querying. This could perhaps be exposed as composite policy type. - [ ] Explore allowing enrich processor to query source index directly. In cases where flexibility is more important than performance. #73407 - [ ] Add more surface area for stats collection on enrich indices, including average search times, total search operations, and average size of search operations.

Explore allowing multiple match fields and a combination of policy types. To allow more dynamic querying. This could perhaps be exposed as composite policy type.

BenB196 · February 25, 2023, 2:41pm

This isn't directly possible, but a solution might be:

Create an Enrichment Policy whose match_field is affected_version.
- An assumption here is that affected_version probably will produce the least number of duplicate matches for step 2
Add an enrich processor and set max_matches to a value greater than 1.
- The exact number to set here is somewhat hard to determine, but you'd want it to be high enough to almost always guarantee you'll match the document which contains the desired affected_product. You could possibly just set it to the max of 128, but I'm not sure how this will impact performance.
Have a script processor loop through the output of the enrich processor and compare the values of affected_product, dropping any matches that don't equal.

You should now in theory have a document that is enriched by both affected_version and max_matches. This solution is nowhere near as efficient as a native multi-match enrich processor would be, but it at least allows for it to be doable.

stephenb · February 25, 2023, 4:45pm

You could just concatenate both the fields into a new field in each index when you ingest both indices. Then the enrich would work fine.

Might even be able to do it with a mapped runtime in the lookup index... Hmmm perhaps I will test that later today

stephenb · February 25, 2023, 5:35pm

Oh and @Sebastian_Huettersen and @maggo Welcome to the community!!

Sebastian_Huettersen · February 25, 2023, 7:06pm

Here's my thought approach if anyone else wants to implement something similar.

With the oval data you quickly reach the limit of max match 128, if you don't change anything directly on the oval data. Therefore, I would recommend merging the oval IDs to:

os_vendor, os_version,product_name, fixed_version.

This way you can reduce the number of oval documents.

Next you have to create a possibility to create a match between the osquery data and the oval data. Here I would use a set processor in the ingest pipeline that combines host.os.codename and product name.

Versions we can not match because all versions in a certain version range are affected not only 1 version. If we in the current state could combine different policy-type 's like match and range that shouldent work in the most cases since the version nr. cannot depict as a range.

Next, create an enrich process with "max_matches" : 128 and as field we can use the new created field with the combined value of OS and product name for matching.

Now comes the trick of a script processor with painless.

"script": {
          "description": "Check if Version still affected",
          "lang": "painless",
          "source": """
              // Method to determined if package is affected
              int checkPackage(){
                //Do magic
              }

              if(some checks if fields exist and have correct datatype){
                for (element in ctx['enrich_taged_field']){
                  if (checkPackage() < 0){
                    //Copy info to target field
                  }
                }
              }
          """
}

At the end remove the target_field from the enrich processor.

@stephenb would be nice there would be more examples in the documentation about enrich especialy templates to bypass the current limentation from enrich.

stephenb · February 25, 2023, 7:46pm

Agree, Feel Free to Open an Enhancement Request against the Documents and perhaps provide some sample use cases...

I am seeing more of these with CVE databases perhaps a feature request around CVE Matching Against the Elasticsearch Repository and tag is Security.

I am not familiar in detail with the OVAL database so a few things are unclear to me but here is an

I do see the new JSON format like this so perhaps where the range would make sense?

                "version": "1.0.0",
                "status": "affected",
                "lessThan": "1.0.6",

Example based on the OP @maggo question he did not specify ranges so I am not sure if that applies or not.

Here is some simple code, I am sure it will not solve the entire issue but perhaps it will provide some other ideas.

It does some processing when loading the CVE data
Then it uses similar processing before calling the enrich processor

DELETE discuss-cve-data

PUT /discuss-cve-data
{
  "mappings": {
    "properties": {
      "CVEID": {
        "type": "keyword"
      },
      "affected_product": {
        "type": "keyword"
      },
      "affected_version": {
        "type": "keyword"
      },
      "notes": {
        "type": "keyword"
      },
      "cve_product_version_unique_id": {
        "type": "keyword"
      }
    }
  }
}


PUT _ingest/pipeline/discuss-cve-prep
{
  "processors": [
    {
      "set": {
        "field": "cve_product_version_unique_id",
        "value": "{{affected_product}} - {{affected_version}}"
      }
    }
  ]
}


POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "chrome",
  "affected_version": "1.5",
  "CVEID": "CVE-1234-5678",
  "notes" : "Bad Chrome Bug"
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "chrome",
  "affected_version": "1.5",
  "CVEID": "CVE-1234-3333",
  "notes" : "Another Bad Chrome Bug"
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "chrome",
  "affected_version": "1.6",
  "CVEID": "CVE-1234-9999",
  "notes" : "Worse Chrome Bug"
}

POST discuss-cve-data/_doc?pipeline=discuss-cve-prep
{
  "affected_product": "safari",
  "affected_version": "3.2",
  "CVEID": "CVE-1234-8888",
  "notes" : "Bad Safari Bug"
}

GET discuss-cve-data/_search
{
  "fields": ["*"]
}

DELETE /_enrich/policy/discuss-cve-data-policy

# Create enrichment policy
PUT /_enrich/policy/discuss-cve-data-policy
{
	"match": {
		"indices": "discuss-cve-data",
		"match_field": "cve_product_version_unique_id",
		"enrich_fields": ["cveid","notes"]
	}
}

POST /_enrich/policy/discuss-cve-data-policy/_execute

DELETE _ingest/pipeline/discuss-cve-lookup

PUT _ingest/pipeline/discuss-cve-lookup
{
  "processors": [
{
      "set": {
        "field": "cve_product_version_unique_id",
        "value": "{{product}} - {{version}}"
      }
    },
    {
      "enrich": {
        "policy_name": "discuss-cve-data-policy",
        "field": "cve_product_version_unique_id",
        "target_field": "cve_data",
        "max_matches": 128
      }
    }
  ]
}

DELETE discuss-product-logs

PUT /discuss-product-logs
{
  "mappings": {
    "properties": {
      "product": {
        "type": "keyword"
      },
      "version": {
        "type": "keyword"
      },
      "message": {
        "type": "keyword"
      },
      "cve_product_version_unique_id": {
        "type": "keyword"
      }
    }
  }
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "chrome",
  "version": "1.5",
  "message": "this is a log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "chrome",
  "version": "1.6",
  "message": "this is another log line"
}


POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "chrome",
  "version": "1.7",
  "message": "this is another log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "safari",
  "version": "3.2",
  "message": "this is another log line"
}

POST discuss-product-logs/_doc?pipeline=discuss-cve-lookup
{
  "product": "safari",
  "version": "3.3",
  "message": "this is another log line"
}

GET discuss-product-logs/_search

Results

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "discuss-product-logs",
        "_id": "nDcaioYBQV01Dpx0-vsY",
        "_score": 1,
        "_source": {
          "cve_data": [
            {
              "cve_product_version_unique_id": "chrome - 1.5",
              "notes": "Bad Chrome Bug"
            },
            {
              "cve_product_version_unique_id": "chrome - 1.5",
              "notes": "Another Bad Chrome Bug"
            }
          ],
          "product": "chrome",
          "message": "this is a log line",
          "version": "1.5",
          "cve_product_version_unique_id": "chrome - 1.5"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "nTcaioYBQV01Dpx0-vs_",
        "_score": 1,
        "_source": {
          "cve_data": [
            {
              "cve_product_version_unique_id": "chrome - 1.6",
              "notes": "Worse Chrome Bug"
            }
          ],
          "product": "chrome",
          "message": "this is another log line",
          "version": "1.6",
          "cve_product_version_unique_id": "chrome - 1.6"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "njcaioYBQV01Dpx0-vtL",
        "_score": 1,
        "_source": {
          "product": "chrome",
          "message": "this is another log line",
          "version": "1.7",
          "cve_product_version_unique_id": "chrome - 1.7"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "nzcaioYBQV01Dpx0-vtX",
        "_score": 1,
        "_source": {
          "cve_data": [
            {
              "cve_product_version_unique_id": "safari - 3.2",
              "notes": "Bad Safari Bug"
            }
          ],
          "product": "safari",
          "message": "this is another log line",
          "version": "3.2",
          "cve_product_version_unique_id": "safari - 3.2"
        }
      },
      {
        "_index": "discuss-product-logs",
        "_id": "oDcaioYBQV01Dpx0-vtj",
        "_score": 1,
        "_source": {
          "product": "safari",
          "message": "this is another log line",
          "version": "3.3",
          "cve_product_version_unique_id": "safari - 3.3"
        }
      }
    ]
  }
}

maggo · February 27, 2023, 2:32pm

Hello stephenb,
thanks for your help!

I'm working together with sebastian on this implementation. We figured out a few things.
We could find a working solution to enrich osquery data with the affected oval data.

We need more time for testing and will come back later.

Kind regards,
Marco

maggo · February 27, 2023, 2:35pm

Hello BenB196,
thanks for your help.

Yes that was one first try to get the 1:1 version match running. After that we recognized that older versions of the application are also affected. So we implemented a version comparison as painless script processor. We need more testing on the script right now.

Have a nice day!
Marco

system · March 27, 2023, 2:35pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Enrich processor is sometimes not enriching Elasticsearch ingest-pipeline	2	393	January 13, 2023
Enrichment Policy does not match the values of an array/object Elasticsearch ingest-pipeline	5	503	November 2, 2022
Use Enrich policy and enrich pipeline processor to check secondary index and update a value Elasticsearch ingest-pipeline	2	575	March 22, 2021
Using Ingest node Enrich your data with "multiple" match fields Elasticsearch	1	1362	January 7, 2020
Disable adding match_field when enriching data in ingest pipeline? Elasticsearch ingest-pipeline , reindex	2	428	October 9, 2022

Elastic enrich data based on two matching fields

Related topics