Enrich processor high cpu load

Hi @leandrojmp Thanks for the detail.

At this point I will just be "Armchair Architect", I am still not clear why enrich is so much CPU.
Overwriting the field event.code does seem a bit weird to me.

I suggest what works for you ... if that is the set processor approach so be it.

Perhaps and yes I have used them successfully when I have CPU intensive ingest pipelines, below is just a perspective not a prescription :slight_smile:

So if you set up ingest nodes yes you will point Filebeat / Logstash etc to the ingest node. The ingest pipeline which is executed pre-write is executed on the ingest node then the ingest node will send the write to where you have the routing allocation set.

For Self Managed Licensing:
Ingest and Coordinator nodes are not licensed.
Licensed nodes include. Master, Data, ML, CCS Nodes
Non Licensed: Ingest / Coordinator Only Nodes, Kibana, Logstash, Beats, Agents, APM Server etc.

Also you should remove the ingest role from your warm and hot nodes then. (technically you should remove ingest from your warm anyways)

An ingest Node (folks often use them as Coordinators as well i.e. Pointing Kibana and Queries to them) so as to leave the Data Nodes to only Read / Write... maximize your licensed nodes.

You could try to set up 2 Ingest Nodes say 4CPU 8GB or so and give it a try you might find it helps (probably will) BUT that is more to manage etc... etc..

Perhaps take a look here

Prerequisites

  • Nodes with the ingest node role handle pipeline processing. To use ingest pipelines, your cluster must have at least one node with the ingest role. For heavy ingest loads, we recommend creating dedicated ingest nodes.