Enrichment doesn't work sometimes

Elastic Stack Elasticsearch
1

Currently we have a situation where the enrichment processor of the elasticsearch ingest pipeline
doesn't always work.

Elastic-Stack: 8.8.1

  • The syslog messages have the identical structure and are parsed correctly. I've manually verified this information with the elasticsearch ingest pipeline editor.

  • The load of the systems ingesting are around 10% CPU and the nodes do have 128 GB Memory. There is no indicator of memory pressure or high Java Heap usage.

  • The Ingest Nodes have the ingest role and data_content roles assigned.

  • The enrichment Pipeline and the enrichment index were not modified in any way while ingesting the data.

  • The ingest pipeline does have about 80 Processors assigned

  • The enrichment index contains 11 fields that are enriched to the documents

  • The field I'm using for linking the enrichment is an ip address

  • The field for the linking to the enrichment processor exists only once in the enrichment index

I'm glad for help to debug this further.

My wild guess is:

  • Some kind of overload of the ingest pipeline/cache and missing enrichments due to some upper time limits for enrichment process itself
2

Can you elaborate more on what you are seeing to make this determination?

3

Can you share an example of how your document looks like and a document that should've been enriched but wasn't?

Also, share your enrich policy and how the data looks like in your source index.

If possible share your entire ingest pipeline as well, so it is possible to try to replicate the issue.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.