# Problems with enrichments

**URL:** <https://discuss.elastic.co/t/problems-with-enrichments/374215>\
**Category:** Elastic Search\
**Created:** [February 7, 2025, 9:46am UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215 "2025-02-07T09:46:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 9:46am UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/1 "2025-02-07T09:46:04Z")

</div>

Hello,

The process of working with enrichments has been quite challenging for me, and I've been struggling with it for hours. It's first time so I may have done something wrong. However, I believe I'm close to a solution, though I’m still having difficulty finding something that works.

I have two dynamically generated fields from the MITRE ATT&CK framework:

attack\_technique: "T1027.010"  
attack\_tactic: "TA0005"  
I would like to enrich these fields to display their full names for better readability. For example:

TA0005 → "Defense Evasion"  
T1027.010 → "Command Obfuscation"  
I have already ingested the MITRE framework in JSON format. However, when I check Analytics → Discover in the logs, I cannot find fields like attack\_tactic\_description or attack\_technique\_description.

Below is my current configuration. Could you please help me identify what might be missing or what I could improve?

Thanks in advance!

```auto
#######################################
Bulk Indexing: Mitre Framework into json.

jq -c '. | {"index": {"_index": "mitre_mapping"}}, .' tactics.json > tactics_bulk.json
jq -c '. | {"index": {"_index": "mitre_mapping"}}, .' techniques.json > techniques_bulk.json

curl -X POST "https://x:9200/_bulk" -H "Content-Type: application/json" --data-binary "@tactics_bulk.json" -u elastic:xxx -k
curl -X POST "https://x:9200/_bulk" -H "Content-Type: application/json" --data-binary "@techniques_bulk.json" -u elastic:xxx -k
#######################################

#######################################
Filebeat Configuration:

output.elasticsearch.indices:
  - index: "carbon_black_observations-%{+yyyy.MM.dd}"
    pipeline: "mitre_attack_pipeline"

#######################################

Index Mapping:

PUT mitre_mapping
{
  "mappings": {
    "properties": {
      "attack_tactic": {
        "type": "keyword"
      },
      "attack_tactic_description": {
        "type": "text"
      },
      "attack_technique": {
        "type": "keyword"
      },
      "attack_technique_description": {
        "type": "text"
      }
    }
  }
}

#######################################

#######################################
Ingest Pipelines: mitre_attack_pipeline

[
  {
    "enrich": {
      "policy_name": "mitre_tactic_policy",
      "field": "attack_tactic",
      "target_field": "attack_tactic_description",
      "ignore_missing": true
    }
  },
  {
    "enrich": {
      "policy_name": "mitre_technique_policy",
      "field": "attack_technique",
      "target_field": "attack_technique_description",
      "ignore_missing": true
    }
  }
]

#######################################
#######################################
Enrich Policies:

mitre_tactic_policy
{
  "policies": [
    {
      "config": {
        "match": {
          "name": "mitre_tactic_policy",
          "indices": [
            "mitre_mapping"
          ],
          "match_field": "attack_tactic",
          "enrich_fields": [
            "attack_tactic_description"
          ]
        }
      }
    }
  ]
}

mitre_technique_policy
{
  "policies": [
    {
      "config": {
        "match": {
          "name": "mitre_technique_policy",
          "indices": [
            "mitre_mapping"
          ],
          "match_field": "attack_technique",
          "enrich_fields": [
            "attack_technique_description"
          ]
        }
      }
    }
  ]
}

#######################################
#######################################
Execute Enrich Policies
POST _enrich/policy/mitre_tactic_policy/_execute
POST _enrich/policy/mitre_technique_policy/_execute

#######################################
#######################################

Testing pipleing:
Which looks like it work

POST _ingest/pipeline/mitre_attack_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "attack_tactic": "TA0005",
        "attack_technique": "T1027.010"
      }
    }
  ]
}

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "attack_tactic": "TA0005",
          "attack_tactic_description": {
            "attack_tactic_description": "Defense Evasion",
            "attack_tactic": "TA0005"
          },
          "attack_technique": "T1027.010",
          "attack_technique_description": {
            "attack_technique_description": "Command Obfuscation",
            "attack_technique": "T1027.010"
          }
        },
        "_ingest": {
          "timestamp": "2025-02-04T13:28:23.091222826Z"
        }
      }
    }
  ]
}

#######################################

```

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [February 7, 2025, 2:46pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/2 "2025-02-07T14:46:59Z")

</div>

Hi @arcsons,

Thanks for sharing the details of your enrichment. Have your documents for enrichment been ingested after you have created and applied your policy [as covered in these steps](https://www.elastic.co/guide/en/elasticsearch/reference/current/match-enrich-policy-type.html)? I see a simulation example in the provided steps but not confirmation of a document being ingested once the ingest pipeline has been setup.

Let us know!

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 4:01pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/3 "2025-02-07T16:01:53Z")

</div>

Hey Carly,  
Yes it has been new logs 😕 Also I can't find the new fields.  
Should be something I have forgot.. Any idea how can I troubleshoot further?

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [February 7, 2025, 5:13pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/4 "2025-02-07T17:13:36Z")

</div>

Do you have any errors coming through the ingest pipeline at all?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 5:42pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/5 "2025-02-07T17:42:32Z")

</div>

> [@arcsons](#):
>
> Hey Carly,  
> Yes it has been new logs 😕 Also I can't find the new fields.  
> Should be something I have forgot.. Any idea how can I troubleshoot further?

How do you know your pipeline is even being called / invoke?...

you can add a `set` processor and add field like `pipeline_name` and set the pipeline name so you know that the pipeline is actually being called.

How are you invoking the pipeline?  
What version of the stack are you on?

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 6:16pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/6 "2025-02-07T18:16:54Z")

</div>

> [@carly.richmond](#):
>
> Do you have any errors coming through the ingest pipeline at all?

I'm not sure to how check enrichment pipeline logs.

GET \_nodes/stats/ingest

Give me something like:

```
  "mitre_attack_pipeline": {
        "count": 48,
        "time_in_millis": 33,
        "current": 0,
        "failed": 24,
        "ingested_as_first_pipeline_in_bytes": 0,
        "produced_as_first_pipeline_in_bytes": 0,
        "processors": [
          {
            "enrich": {
              "type": "enrich",
              "stats": {
                "count": 22,
                "time_in_millis": 4,
                "current": 0,
                "failed": 0
              }
            }
          },

```

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 6:22pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/7 "2025-02-07T18:22:59Z")

</div>

I'm a beginner of Elastic so I dont know if its called or not...

Add processor, do you mean something like this?

```auto
PUT _ingest/pipeline/mitre_attack_pipeline
{
  "description": "Pipeline to enrich MITRE ATT&CK fields",
  "processors": [
    {
      "set": {
        "field": "pipeline_name",
        "value": "mitre_attack_pipeline"
      }
    },
    {
      "enrich": {
        "policy_name": "mitre_tactic_policy",
        "field": "attack_tactic",
        "target_field": "attack_tactic_description",
        "ignore_missing": true
      }
    },
    {
      "enrich": {
        "policy_name": "mitre_technique_policy",
        "field": "attack_technique",
        "target_field": "attack_technique_description",
        "ignore_missing": true
      }
    }
  ]
}

```

Invoking the pipeline do you mean i filebeat.yml:  
output.elasticsearch.indices:

- index: "carbon\_black\_observations-%{+yyyy.MM.dd}"  
pipeline: "mitre\_attack\_pipeline"

Eleastic version is: 8.15.3

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 6:25pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/8 "2025-02-07T18:25:02Z")

</div>

so when you add the set processor the field `pipeline_name` in your logs now? if not the pipeline is not getting called.

Do you have logstash in the middle for direct filebeat -\> elasticsearch?

Can you share the rest of your filebeat.yml?

Perhaps try this .. .i think `pipeline` is under the wrong level I don't think it is getting called `pipeline` is at the top level of `output.elasticsearch`

```auto
output.elasticsearch.indices:
  index: "carbon_black_observations-%{+yyyy.MM.dd}"
output.elasticsearch.pipeline: "mitre_attack_pipeline"

```

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 6:39pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/9 "2025-02-07T18:39:53Z")

</div>

FIlebeat -\> Elastic.

This gave me error when I restarted filebeat.

```auto
output.elasticsearch.indices:
  - index: "carbon_black_observations-%{+yyyy.MM.dd}"
# pipeline: "mitre_attack_pipeline"
    when.contains:
      event.dataset: "carbon_black.observations"
output.elasticsearch.pipeline: "mitre_attack_pipeline"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 6:42pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/10 "2025-02-07T18:42:01Z")

</div>

> [@arcsons](#):
>
> This gave me error when I restarted filebeat.

Exactly what error... can not help without details

please share the entire filebeat.yml and the entire error otherwise we are looking through a pin hole at an elephant 🙂

Perhaps review the docs...

> **[Configure the Elasticsearch output | Filebeat Reference \[8.15\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/8.15/elasticsearch-output.html)**

are you trying to set a pipeline per input or per index ?

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 7:26pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/11 "2025-02-07T19:26:30Z")

</div>

sudo systemctl status filebeat  
× filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.  
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; preset: enabled)  
Active: failed (Result: exit-code) since Fri 2025-02-07 19:35:04 CET; 37s ago  
Duration: 68ms  
Docs: [Filebeat: Lightweight Log Analysis & Elasticsearch | Elastic](https://www.elastic.co/beats/filebeat)  
Process: 1791384 ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT\_LOG\_OPTS $BEAT\_CONFIG\_OPTS $BEAT\_PATH\_OPTS (code=exited, status=1/FAILURE)  
Main PID: 1791384 (code=exited, status=1/FAILURE)  
CPU: 77ms

Feb 07 19:35:04 x.arc.local systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.  
Feb 07 19:35:04 x.arc.local systemd[1]: filebeat.service: Start request repeated too quickly.  
Feb 07 19:35:04 x.arc.local systemd[1]: filebeat.service: Failed with result 'exit-code'.  
Feb 07 19:35:04 x.arc.local systemd[1]: Failed to start filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch..

```auto
cat /etc/filebeat/filebeat.yml
setup.template.name: "carbon_black"
setup.template.pattern: "carbon_black-*"
setup.template.enable: false
setup.ilm.enabled: false

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/arc/cb.json
  json.keys_under_root: true
  json.overwrite_keys: true
  processors:
    - fingerprint:
        fields: ["event_id", "device_timestamp"]
        target_field: "@metadata._id"
    - add_fields:
        target: event
        fields:
          dataset: "carbon_black.observations"

- type: log
  enabled: true
  paths:
   - /home/arc/vuln.json
  json.keys_under_root: true
  json.overwrite_keys: true
  processors:
    - fingerprint:
        fields: ["created_at", "cve_id"]
        target_field: "@metadata._id"
    - add_fields:
        target: event
        fields:
          dataset: "carbon_black.vulnerabilities"

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.elasticsearch:
  hosts: ["https://x:9200", "https://x:9200", "https://x:9200"]
  protocol: "https"
  ssl.verification_mode: "none"
  username: "elastic"
  password: "xxxx"

output.elasticsearch.indices:
  - index: "carbon_black_observations-%{+yyyy.MM.dd}"
# pipeline: "mitre_attack_pipeline"
    when.contains:
      event.dataset: "carbon_black.observations"
output.elasticsearch.pipeline: "mitre_attack_pipeline"

  - index: "carbon_black_vulnerabilities-%{+yyyy.MM.dd}"
    when.contains:
      event.dataset: "carbon_black.vulnerabilities"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640
´´´
```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 8:31pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/12 "2025-02-07T20:31:45Z")

</div>

Mostly Likely you have .yml errors

run

`/usr/share/filebeat/bin/filebeat -c /etc/filebeat.yml test config`

logs... what shown is not enough

`journalctl -u filebeat.service`

Logs also may be `/var/log/filebeat`

Should look like

```auto
output.elasticsearch:
  hosts: ["https://x:9200", "https://x:9200", "https://x:9200"]
  protocol: "https"
  ssl.verification_mode: "none"
  username: "elastic"
  password: "xxxx"
  pipeline: "mitre_attack_pipeline"

```

take out  
`output.elasticsearch.pipeline: "mitre_attack_pipeline"`

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 8:43pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/13 "2025-02-07T20:43:58Z")

</div>

> [@stephenb](#):
>
> ```auto
> output.elasticsearch:
> hosts: ["https://x:9200", "https://x:9200", "https://x:9200"]
> protocol: "https"
> ssl.verification_mode: "none"
> username: "elastic"
> password: "xxxx"
> pipeline: "mitre_attack_pipeline"
> 
> ```

Yes that was the original config I had before you said I should try with:  
output.elasticsearch.pipeline: "mitre\_attack\_pipeline"

output.elasticsearch:  
hosts: ["[https://x:9200](https://x:9200)", "[https://x:9200](https://x:9200)", "[https://x:9200](https://x:9200)"]  
protocol: "https"  
ssl.verification\_mode: "none"  
username: "elastic"  
password: "xxxx"  
pipeline: "mitre\_attack\_pipeline"

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 7, 2025, 8:45pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/14 "2025-02-07T20:45:08Z")

</div>

Also, if the filebeat service is not starting, you won't get the error on filebeat logs, you need to look in the system logs, `/var/log/messages` or `/var/log/syslog`.

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 8:47pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/15 "2025-02-07T20:47:08Z")

</div>

I have this now and it has no errors when i start it.

output.elasticsearch:  
hosts: ["[https://x:9200](https://x:9200)", "[https://x:9200](https://x:9200)", "[https://x:9200](https://x:9200)"]  
protocol: "https"  
ssl.verification\_mode: "none"  
username: "elastic"  
password: "xxx"

output.elasticsearch.indices:

- index: "carbon\_black\_observations-%{+yyyy.MM.dd}"  
pipeline: "mitre\_attack\_pipeline"  
when.contains:  
event.dataset: "carbon\_black.observations"

I have tested with add pipeline: "mitre\_attack\_pipeline" under output.elasticsearch without luck.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 9:41pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/16 "2025-02-07T21:41:15Z")

</div>

Put it back the Way I showed you.

And run the command I gave you to run

`/usr/share/filebeat/bin/filebeat -c /etc/filebeat.yml test config`

also run

`/usr/share/filebeat/bin/filebeat -c /etc/filebeat.yml test output`

Or just run the entire run command in the foreground

`/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat.yml`

You will see the actual error in one of these

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 9:57pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/17 "2025-02-07T21:57:46Z")

</div>

I have no error for start filebeat. Its the enirchment I have problems with 🙂

$ sudo /usr/bin/filebeat -c /etc/filebeat/filebeat.yml test config  
Config OK

sudo /usr/bin/filebeat -c /etc/filebeat/filebeat.yml test output  
elasticsearch: [https://x:9200](https://x:9200)...  
parse url... OK  
connection...  
parse host... OK  
dns lookup... OK  
addresses: x  
dial up... OK  
TLS...  
security... WARN server's certificate chain verification is disabled  
handshake... OK  
TLS version: TLSv1.3  
dial up... OK  
talk to server... OK  
version: 8.15.3  
elasticsearch: [https://x:9200](https://x:9200)...  
parse url... OK  
connection...  
parse host... OK  
dns lookup... OK  
addresses: x  
dial up... OK  
TLS...  
security... WARN server's certificate chain verification is disabled  
handshake... OK  
TLS version: TLSv1.3  
dial up... OK  
talk to server... OK  
version: 8.15.3  
elasticsearch: [https://x:9200](https://x:9200)...  
parse url... OK  
connection...  
parse host... OK  
dns lookup... OK  
addresses: x  
dial up... OK  
TLS...  
security... WARN server's certificate chain verification is disabled  
handshake... OK  
TLS version: TLSv1.3  
dial up... OK  
talk to server... OK  
version: 8.15.3

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 10:05pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/18 "2025-02-07T22:05:30Z")

</div>

Good now start in the foreground

> [@stephenb](#):
>
> /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat.yml

Look for errors.

If no errors Go to Discover and look for documents

They should at the very least have the field you added with the set processorThey should at the very least have the fields you added with the set processor

---

<div class="post-metadata">

**Author:** ![arcsons](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@arcsons](https://discuss.elastic.co/u/arcsons)\
**Post date:** [February 7, 2025, 10:13pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/19 "2025-02-07T22:13:58Z")

</div>

Sounds like a nice plan, started it now. Lets wait for a new log entry to come. Thanks for helping really nice!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2025, 10:23pm UTC](https://discuss.elastic.co/t/problems-with-enrichments/374215/20 "2025-02-07T22:23:03Z")

</div>

Keep and eye on the filebeat logs should be printing to stdout...

[Next page](https://discuss.elastic.co/t/problems-with-enrichments/374215.md?page=2)