Enrichment Policy does not match the values of an array/object

hanna · September 30, 2022, 1:25pm

Hi, i have the following enrichmentpolicy that i cant get to work.
The targetfield cveintel from the enrichmentpolicy is never created.
Any help is appreciated.

> PUT _enrich/policy/cve_enrichment_policy
> {
>   "match": {
>     "indices": "logstash-knowncves*",
>     "match_field": "cveID",
>     "enrich_fields": ["event.original"]
>   }
> }

>  POST _enrich/policy/cve_enrichment_policy/_execute

PUT _ingest/pipeline/openvas_enrichment
{
  "processors": [
    {
      "enrich": {
        "field": "nvt.refs.ref.@id.keyword",
        "policy_name": "cve_enrichment_policy",
        "target_field": "cveintel",
        "ignore_missing": true,
        "max_matches": 128
      }
    }
  ]
}

Mapping of destination index that should get enriched:

PUT /logstash-jsonfiles-2022.09.29/_mapping
{
      "dynamic_templates": [
        {
          "message_field": {
            "path_match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "norms": false,
              "type": "text"
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "norms": false,
              "type": "text"
            }
          }
        }
      ],
      "properties": {
            "refs": {
              "properties": {
                "ref": {
                  "properties": {
                    "@id": {
                      "type": "text",
                      "norms": false
                    },
                    "@type": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "norms": false
                    }
                  }
                }
              }
            }
        }
      }
    }
  }

Posting my document against this pipeline does not enrich the target field.

> POST /_ingest/pipeline/openvas_enrichment/_simulate?verbose=true
> {
>   "docs": [ 
>       {
>         "_index": "logstash-jsonfiles-2022.09.29",
>         "_id": "n7tHioMBzhwZrgzjzCmt",
>         "_score": 1,
>         "_ignored": [
>           "nvt.tags.keyword",
>           "event.original.keyword"
>         ],
>         "_source": {
>           "owner": {
>             "name": "redacted"
>           },
>           "creation_time": "2022-07-10T23:08:11Z",
>           "severity": "9.8",
>           "original_severity": "9.8",
>           "log": {
>             "file": {
>               "path": "/usr/share/jsonfiles/sfsfd.ndjson"
>             }
>           },
>           "qod": {
>             "type": null,
>             "value": "80"
>           },
>           "description": """Installed version: 99.2.2.1
> Fixed version:     Contact vendor""",
>           "scan_nvt_version": "2021-09-29T10:01:38Z",
>           "nvt": {
>             "@oid": "1.3.6.1.4.1.25623.1.0.146309",
>             "solution": {
>               "#text": "Contact the vendor for a solution.",
>               "@type": "VendorFix"
>             },
>             "refs": {
>               "ref": [
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2013-6282"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2018-12326"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2018-11218"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2020-4670"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2018-8014"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-33020"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2018-10115"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-27501"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-33018"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-27497"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2012-1708"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2015-9251"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-27493"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2019-9636"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-33024"
>                 },
>                 {
>                   "@type": "cve",
>                   "@id": "CVE-2021-33022"
>                 },
>                 {
>                   "@type": "cisa",
>                   "@id": "Known Exploited Vulnerability (KEV) catalog"
>                 },
>                 {
>                   "@type": "url",
>                   "@id": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
>                 },
>                 {
>                   "@type": "url",
>                   "@id": "https://us-cert.cisa.gov/ics/advisories/icsma-21-187-01"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K22/0045"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/1030"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0711"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0705"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0693"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0555"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0543"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0309"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0154"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K20/0041"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0911"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0909"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0615"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0321"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0313"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0054"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0052"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0049"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0048"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K19/0046"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K18/1006"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K18/1005"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K18/0680"
>                 },
>                 {
>                   "@type": "cert-bund",
>                   "@id": "CB-K18/0647"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2021-1736"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-2423"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-2130"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1839"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1540"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1508"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1413"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1276"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1134"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-1078"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0850"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0835"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0821"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0630"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0590"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0569"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0557"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0501"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0381"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0318"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0231"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0177"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0111"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0102"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2020-0048"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-2710"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-2252"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-2158"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-2078"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1877"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1704"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1627"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1537"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1455"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1339"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1288"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1285"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1237"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-1095"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0941"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0915"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0912"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0841"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0777"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0772"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0770"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0702"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0565"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0451"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0119"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2019-0111"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-2474"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-2194"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-2165"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-2142"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-2110"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-2103"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1674"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1443"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1416"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1253"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1163"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1153"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-1038"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-0939"
>                 },
>                 {
>                   "@type": "dfn-cert",
>                   "@id": "DFN-CERT-2018-0853"
>                 }
>               ]
>             },
>             "name": "Philips Vue PACS Multiple Vulnerabilities (Jul 2021)",
>             "cvss_base": "9.8",
>             "type": "nvt",
>             "family": "General",
>             "tags": """cvss_base_vector=CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H|summary=Philips Vue PACS is prone to multiple vulnerabilities.|insight=The following vulnerabilities exist:
> 
>   - CVE-2020-1938: The product receives input or data, but it does not validate or incorrectly
>   validates that the input has the properties that are required to process the data safely and
>   correctly.
> 
>   - CVE-2018-12326, CVE-2018-11218: The software performs operations on a memory buffer, but it can
>   read from or write to a memory location that is outside of the intended boundary of the buffer.
>   This vulnerability exists within a third party software component (Redis).
> 
>   - CVE-2020-4670: When an actor claims to have a given identity, the software does not prove or
>   insufficiently proves that the claim is correct. This vulnerability exists within a third party
>   software component (Redis).
> 
>   - CVE-2018-8014: The software initializes or sets a resource with a default that is intended to
>   be changed by the administrator, but the default is not secure.
> 
>   - CVE-2021-33020: The product uses a cryptographic key or password past its expiration date,
>   which diminishes its safety significantly by increasing the timing window for cracking attacks
>   against that key.
> 
>   - CVE-2018-10115: The software does not initialize or incorrectly initializes a resource, which
>   might leave the resource in an unexpected state when it is accessed or used. This vulnerability
>   exists within a third party software component (7-Zip).
> 
>   - CVE-2021-27501: The software does not follow certain coding rules for development, which can
>   lead to resultant weaknesses or increase the severity of the associated vulnerabilities.
> 
>   - CVE-2021-33018: The use of a broken or risky cryptographic algorithm is an unnecessary risk
>   that may result in the exposure of sensitive information.
> 
>   - CVE-2021-27497: The product does not use or incorrectly uses a protection mechanism that
>   provides sufficient defense against directed attacks against the product.
> 
>   - CVE-2012-1708: Weaknesses in this category is related to a software system's data integrity
>   components. This vulnerability exists within a third party software component (Oracle Database).
> 
>   - CVE-2015-9251: The software does not neutralize or incorrectly neutralizes user-controllable
>   input before it is placed in an output used as a webpage that is served to other users.
> 
>   - CVE-2021-27493: The product does not ensure or incorrectly ensures structured messages or data
>   are well formed and that certain security properties are met before being read from an upstream
>   component or sent to a downstream component.
> 
>   - CVE-2019-9636: The software does not properly handle when an input contains Unicode encoding.
> 
>   - CVE-2021-33024: The product transmits or stores authentication credentials, but it uses an
>   insecure method susceptible to unauthorized interception and/or retrieval.
> 
>   - CVE-2021-33022: The software transmits sensitive or security-critical data in cleartext in a
>   communication channel that can be sniffed by unauthorized actors.|affected=Philips Vue PACS version 12.2.x.x and prior.|impact=|solution=Contact the vendor for a solution.|vuldetect=Checks if a vulnerable version is present on the target host.|solution_type=VendorFix""",
>             "severities": {
>               "severity": {
>                 "date": "2021-02-24T12:15:00Z",
>                 "score": "9.8",
>                 "@type": "cvss_base_v3",
>                 "origin": "NVD",
>                 "value": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"
>               },
>               "@score": "9.8"
>             }
>           },
>           "original_threat": "High",
>           "@timestamp": "2022-09-29T17:25:04.483591221Z",
>           "port": "443/tcp",
>           "@version": "1",
>           "host": {
>             "hostname": null,
>             "#text": "172.16.15.31",
>             "name": "logstash1",
>             "asset": {
>               "@asset_id": "64097e6b-1b81-47d5-b9d3-cea3e669b98e"
>             }
>           },
>           "name": "Philips Vue PACS Multiple Vulnerabilities (Jul 2021)",
>           "comment": null,
>           "@id": "0f667a51-5e2b-4155-9639-45756asd",
>           "threat": "High",
>           "event": {
>             "original": """{"@id": "0f667a51-5e2b-4155-9639-45756asd", "name": "Philips Vue PACS Multiple Vulnerabilities (Jul 2021)", "owner": {"name": "redacted"}, "modification_time": "2022-07-10T23:08:11Z", "comment": null, "creation_time": "2022-07-10T23:08:11Z", "host": {"asset": {"@asset_id": "64097e6b-1b81-47d5-b9d3-cea3e669basd"}, "hostname": null, "#text": "172.16.15.31"}, "port": "443/tcp", "nvt": {"@oid": "1.3.6.1.4.1.25623.1.0.146309", "type": "nvt", "name": "Philips Vue PACS Multiple Vulnerabilities (Jul 2021)", "family": "General", "cvss_base": "9.8", "severities": {"@score": "9.8", "severity": {"@type": "cvss_base_v3", "origin": "NVD", "date": "2021-02-24T12:15:00Z", "score": "9.8", "value": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"}}, "tags": "cvss_base_vector=CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H|summary=Philips Vue PACS is prone to multiple vulnerabilities.|insight=The following vulnerabilities exist:\n\n  - CVE-2020-1938: The product receives input or data, but it does not validate or incorrectly\n  validates that the input has the properties that are required to process the data safely and\n  correctly.\n\n  - CVE-2018-12326, CVE-2018-11218: The software performs operations on a memory buffer, but it can\n  read from or write to a memory location that is outside of the intended boundary of the buffer.\n  This vulnerability exists within a third party software component (Redis).\n\n  - CVE-2020-4670: When an actor claims to }"""
>           },
>           "modification_time": "2022-07-10T23:08:11Z"
>         }
>       }
>     ]
> }

Blockquote
What am i doing wrong?

stephenb · September 30, 2022, 7:27pm

Hi @hanna

I think there could be a couple different things going on...

Can you share the mapping for logstash-knowncves*?
Can you share a document 1 or 2 documents from that index?
Finally your source document that you want to enrich has an array of CVE's you want to enrich ...where / how do you expect the result to look like ... as an array under cveintel? Is that your expected result?

I can tell you, this won't work...

  "processors": [
    {
      "enrich": {
        "field": "nvt.refs.ref.@id.keyword", <!--- This wont work in _simulate or real

This will not work with your _simulate because the ingest pipeline works on _source field and there is no nvt.refs.ref.@id.keyword" in your sample document only nvt.refs.ref.@id"

In fact it will not work when actually running because ingest pipeline work before data is written.. before they are turned into fields like keyword or text

If you provide a bit more I might be able to help....

hanna · October 4, 2022, 7:58am

Hi @stephenb

thank you for your help. I have adjusted the enrich processor.
Yes you are right.. i want to enrich preferably all fields from the source indicies (logstash-knowncves*) as array to cveintel.

Here is the Mapping of logstash-knowncves* :

{
  "logstash-knowncves-2022.09.21": {
    "mappings": {
      "dynamic_templates": [
        {
          "message_field": {
            "path_match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "norms": false,
              "type": "text"
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "norms": false,
              "type": "text"
            }
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "keyword"
        },
        "column9": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "cveID": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "dateAdded": {
          "type": "date"
        },
        "dueDate": {
          "type": "date"
        },
        "event": {
          "properties": {
            "original": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "norms": false
            }
          }
        },
        "geoip": {
          "dynamic": "true",
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "half_float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "half_float"
            }
          }
        },
        "host": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "norms": false
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "norms": false
                }
              }
            }
          }
        },
        "message": {
          "type": "text",
          "norms": false
        },
        "product": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "requiredAction": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "shortDescription": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "tags": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "vendorproject": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        },
        "vuln": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "norms": false
        }
      }
    }
  }
}

And here are two documents from a search against that index :

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "logstash-knowncves-2022.09.21",
        "_id": "rH4QYoMBzhwZrgzjwQrm",
        "_score": 1,
        "_ignored": [
          "event.original.keyword"
        ],
        "_source": {
          "log": {
            "file": {
              "path": "/usr/share/threatintel/criticalvulns/known_exploited_cisa.csv"
            }
          },
          "event": {
            "original": "\"CVE-2013-2597\",\"Code Aurora\",\"ACDB Audio Driver\",\"Code Aurora ACDB Audio Driver Stack-based Buffer Overflow Vulnerability\",\"2022-09-15\",\"The Code Aurora audio calibration database (acdb) audio driver contains a stack-based buffer overflow vulnerability which allows for privilege escalation. Code Aurora is used in third-party products such as Qualcomm and Android.\",\"Apply updates per vendor instructions.\",\"2022-10-06\",\"https://web.archive.org/web/20161226013354/https:/www.codeaurora.org/news/security-advisories/stack-based-buffer-overflow-acdb-audio-driver-cve-2013-2597\"\r"
          },
          "host": {
            "name": "logstash1"
          },
          "column9": "https://web.archive.org/web/20161226013354/https:/www.codeaurora.org/news/security-advisories/stack-based-buffer-overflow-acdb-audio-driver-cve-2013-2597",
          "requiredAction": "Apply updates per vendor instructions.",
          "vuln": "Code Aurora ACDB Audio Driver Stack-based Buffer Overflow Vulnerability",
          "product": "ACDB Audio Driver",
          "@version": "1",
          "cveID": "CVE-2013-2597",
          "vendorproject": "Code Aurora",
          "@timestamp": "2022-09-21T22:00:08.575055Z",
          "dueDate": "2022-10-06",
          "dateAdded": "2022-09-15",
          "shortDescription": "The Code Aurora audio calibration database (acdb) audio driver contains a stack-based buffer overflow vulnerability which allows for privilege escalation. Code Aurora is used in third-party products such as Qualcomm and Android."
        },
        "fields": {
          "@timestamp": [
            "2022-09-21T22:00:08.575Z"
          ]
        }
      },
      {
        "_index": "logstash-knowncves-2022.09.21",
        "_id": "_sIQYoMBTL4bbEpdwWfm",
        "_score": 1,
        "_ignored": [
          "event.original.keyword"
        ],
        "_source": {
          "log": {
            "file": {
              "path": "/usr/share/threatintel/criticalvulns/known_exploited_cisa.csv"
            }
          },
          "event": {
            "original": "\"CVE-2013-2596\",\"Linux\",\"Kernel\",\"Linux Kernel Integer Overflow Vulnerability\",\"2022-09-15\",\"Linux kernel fb_mmap function in drivers/video/fbmem.c contains an integer overflow vulnerability which allows for privilege escalation.\",\"Apply updates per vendor instructions.\",\"2022-10-06\",\"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc9bbca8f650e5f738af8806317c0a041a48ae4a\"\r"
          },
          "host": {
            "name": "logstash1"
          },
          "column9": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc9bbca8f650e5f738af8806317c0a041a48ae4a",
          "requiredAction": "Apply updates per vendor instructions.",
          "vuln": "Linux Kernel Integer Overflow Vulnerability",
          "product": "Kernel",
          "@version": "1",
          "cveID": "CVE-2013-2596",
          "vendorproject": "Linux",
          "@timestamp": "2022-09-21T22:00:08.575162Z",
          "dueDate": "2022-10-06",
          "dateAdded": "2022-09-15",
          "shortDescription": "Linux kernel fb_mmap function in drivers/video/fbmem.c contains an integer overflow vulnerability which allows for privilege escalation."
        },
        "fields": {
          "@timestamp": [
            "2022-09-21T22:00:08.575Z"
          ]
        }
      }
    ]
  }
}

Best Regards!

stephenb · October 4, 2022, 11:18pm

@hanna See if you can follow this...

Actually I can't believe I got this to work. I used the foreach processor + enrich.

Unfortunately the enrich fields will need to go inside the Array of CVEs I could not figure out (pretty sure it can not be done) as a separate array somewhere else in the doc... BUT I think that will actually make it easy to get to the data / definitions later as they will be in the same path of the CVE

Its a little shorter version you will need to be carefull with the field names etc.

I used your mapping above for the cve-source

Here you go ... This was cool I learned something

# Put in your mapings for my source index from your logstash-knowncves

# Source / Lookup Doc
POST cve-source/_doc
{
  "log": {
    "file": {
      "path": "/usr/share/threatintel/criticalvulns/known_exploited_cisa.csv"
    }
  },
  "event": {
    "original": "\"CVE-2013-2597\",\"Code Aurora\",\"ACDB Audio Driver\",\"Code Aurora ACDB Audio Driver Stack-based Buffer Overflow Vulnerability\",\"2022-09-15\",\"The Code Aurora audio calibration database (acdb) audio driver contains a stack-based buffer overflow vulnerability which allows for privilege escalation. Code Aurora is used in third-party products such as Qualcomm and Android.\",\"Apply updates per vendor instructions.\",\"2022-10-06\",\"https://web.archive.org/web/20161226013354/https:/www.codeaurora.org/news/security-advisories/stack-based-buffer-overflow-acdb-audio-driver-cve-2013-2597\"\r"
  },
  "host": {
    "name": "logstash1"
  },
  "column9": "https://web.archive.org/web/20161226013354/https:/www.codeaurora.org/news/security-advisories/stack-based-buffer-overflow-acdb-audio-driver-cve-2013-2597",
  "requiredAction": "Apply updates per vendor instructions.",
  "vuln": "Code Aurora ACDB Audio Driver Stack-based Buffer Overflow Vulnerability",
  "product": "ACDB Audio Driver",
  "@version": "1",
  "cveID": "CVE-2013-2597",
  "vendorproject": "Code Aurora",
  "@timestamp": "2022-09-21T22:00:08.575055Z",
  "dueDate": "2022-10-06",
  "dateAdded": "2022-09-15",
  "shortDescription": "The Code Aurora audio calibration database (acdb) audio driver contains a stack-based buffer overflow vulnerability which allows for privilege escalation. Code Aurora is used in third-party products such as Qualcomm and Android."
}

# Source / Lookup Doc
POST cve-source/_doc
{
  "log": {
    "file": {
      "path": "/usr/share/threatintel/criticalvulns/known_exploited_cisa.csv"
    }
  },
  "event": {
    "original": "\"CVE-2013-2596\",\"Linux\",\"Kernel\",\"Linux Kernel Integer Overflow Vulnerability\",\"2022-09-15\",\"Linux kernel fb_mmap function in drivers/video/fbmem.c contains an integer overflow vulnerability which allows for privilege escalation.\",\"Apply updates per vendor instructions.\",\"2022-10-06\",\"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc9bbca8f650e5f738af8806317c0a041a48ae4a\"\r"
  },
  "host": {
    "name": "logstash1"
  },
  "column9": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc9bbca8f650e5f738af8806317c0a041a48ae4a",
  "requiredAction": "Apply updates per vendor instructions.",
  "vuln": "Linux Kernel Integer Overflow Vulnerability",
  "product": "Kernel",
  "@version": "1",
  "cveID": "CVE-2013-2596",
  "vendorproject": "Linux",
  "@timestamp": "2022-09-21T22:00:08.575162Z",
  "dueDate": "2022-10-06",
  "dateAdded": "2022-09-15",
  "shortDescription": "Linux kernel fb_mmap function in drivers/video/fbmem.c contains an integer overflow vulnerability which allows for privilege escalation."
}

# Enrich Policy
PUT _enrich/policy/cve_enrichment_policy
{
  "match": {
    "indices": "cve-source",
    "match_field": "cveID",
    "enrich_fields": [
      "event.original"
    ]
  }
}

# Execute
PUT _enrich/policy/cve_enrichment_policy/_execute

# The pipeline that loops through the array and call enrich
PUT _ingest/pipeline/cve_enrichment_pipeline
{
  "processors": [
    {
      "foreach": {
        "field": "nvt.refs.ref", <!-- The Path to the Array
        "processor": {
          "enrich": {
            "field": "_ingest._value.@id",  <!--- The Field in the array to match 
            "policy_name": "cve_enrichment_policy",
            "target_field": "_ingest._value.cveintel", <!--- The Field in the array to put the enrich data 
            "ignore_missing": true,
            "max_matches": 128
          }
        }
      }
    }
  ]
}

# Simulate
POST _ingest/pipeline/cve_enrichment_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "nvt": {
          "refs": {
            "ref": [
              {
                "@type": "cve",
                "@id": "CVE-2013-2596"
              },
              {
                "@type": "cve",
                "@id": "CVE-2013-2597"
              }
            ]
          }
        }
      }
    }
  ]
}


# Post a doc
POST cve-logs/_doc/?pipeline=cve_enrichment_pipeline
{
  "nvt": {
    "refs": {
      " ref": [
        {
          "@type": "cve",
          "@id": "CVE-2013-2596"
        },
        {
          "@type": "cve",
          "@id": "CVE-2013-2597"
        }
      ]
    }
  }
}

# See the results 
GET cve-logs/_search

# Results

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "cve-logs",
        "_id": "Z4dDpYMBSVkvcWRyi5AU",
        "_score": 1,
        "_ignored": [
          "ref.cveintel.event.original.keyword"
        ],
        "_source": {
          "ref": [
            {
              "@type": "cve",
              "@id": "CVE-2013-2596",
              "cveintel": [
                {
                  "event": {
                    "original": "\"CVE-2013-2596\",\"Linux\",\"Kernel\",\"Linux Kernel Integer Overflow Vulnerability\",\"2022-09-15\",\"Linux kernel fb_mmap function in drivers/video/fbmem.c contains an integer overflow vulnerability which allows for privilege escalation.\",\"Apply updates per vendor instructions.\",\"2022-10-06\",\"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc9bbca8f650e5f738af8806317c0a041a48ae4a\"\r"
                  },
                  "cveID": "CVE-2013-2596"
                }
              ]
            },
            {
              "@type": "cve",
              "@id": "CVE-2013-2597",
              "cveintel": [
                {
                  "event": {
                    "original": "\"CVE-2013-2597\",\"Code Aurora\",\"ACDB Audio Driver\",\"Code Aurora ACDB Audio Driver Stack-based Buffer Overflow Vulnerability\",\"2022-09-15\",\"The Code Aurora audio calibration database (acdb) audio driver contains a stack-based buffer overflow vulnerability which allows for privilege escalation. Code Aurora is used in third-party products such as Qualcomm and Android.\",\"Apply updates per vendor instructions.\",\"2022-10-06\",\"https://web.archive.org/web/20161226013354/https:/www.codeaurora.org/news/security-advisories/stack-based-buffer-overflow-acdb-audio-driver-cve-2013-2597\"\r"
                  },
                  "cveID": "CVE-2013-2597"
                }
              ]
            }
          ]
        }
      }
    ]
  }
}

hanna · October 5, 2022, 8:59am

Stephen you are amazing!

I would never have come up with that, it almost seems like you shared some arcane knowledge here

Thank you very much - you saved me a lot of headaches!

system · November 2, 2022, 9:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Enrich policy not working in logstash pipeline Elasticsearch ingest-pipeline	3	18	October 28, 2024
Elastic enrich data based on two matching fields Elasticsearch ingest-pipeline	9	1940	March 27, 2023
Enrich policy working only on some documents Elasticsearch ingest-pipeline	2	63	July 3, 2024
Enrich processor is sometimes not enriching Elasticsearch ingest-pipeline	2	384	January 13, 2023
Creating a Geo Enrich Policy for existing Index, encountering error when adding the enriched field to existing documents by Reindexing Elasticsearch ingest-pipeline , vector-search	2	30	July 26, 2024

Enrichment Policy does not match the values of an array/object

Related Topics