Enrich processor is sometimes not enriching

[siginigin (sigi)]

Hi,

I have pipeline with enrich processor for user enriching and another pipeline with enrich processor for vulnerability enriching. Both pipelines are in parent logs_syslog pipeline.

I noticed some events are not enriched.
In case of user enrichment I have following policy:

{
  "match": {
    "indices": "ldap",
    "match_field": "user.sam_account_name",
    "enrich_fields": [
      "user.cn", 
      "user.description", 
      "user.object_sid", 
      "user.group", 
      "user.mail",
      "user.principal_name",
      "fingerprint"
    ]
  }
}

Sometimes lot of enrich_fields are missing, for example there is only group added, sometimes no fields, even if the user has all fields and other events are enriched successfully.

In case of vulnerability I have following policy:

{
  "match": {
    "indices": "latest_vulnerabilities",
    "match_field": "server.ip",
    "enrich_fields": [
      "vulnerability.severity_num.max"
    ]
  }
}

Enrich pipeline contains processors for enriching source.ip and destination.ip. In the result, specific source or destination ip is sometimes enriched, sometimes not.

So I think this is pretty simple setup.

I check the enrichment index and it seems it has all relevant data. Testing with _simulate is same, sometimes it returns data, sometimes it doesn't.

I noticed when I POST _execute to policy, I'm able to see the data with "Test pipeline", but after a while (2-3 tests) it will again disappear.

What can be reason of such behavior?

Thank you for help.

[Keith_Massey]

That sounds odd. You say you can reproduce this with the _simulate API -- is it possible to post everything you need in order to do that? I'd like to try to reproduce it.
Also, are there any error messages or stack traces in your Elasticsearch logs?