"The processor action grok does not exist" in FileBeat. Why ? (Custom Logs Integration)

[mehdi-lamrani]

I am a bit confused here.
Grok is available in Ingest Processors but not in Filebeat processors
May I ask why ?

I was hoping to do this

, but it breaks, as I am getting the following error :

[elastic_agent][error] [...] the processor action grok does not exist. 

So,
1 - Why are the 2 processor families different ? why have two sets of processors and not just one ? (philosophical question)
2 - Where should I put my grok then ? (I am using custom logs integration)
My guess is that I need to define an ingest pipeline and add it to custom configurations.
Am I right ?

Thanks in advance,
M.

[rugenl (Len Rugen)]

I think because ingest processor is an ingest pipeline, running on an elasticsearch node. FIlebeat processors are lightweight, limited and run on filebeat. Filebeat processors are good for dropping data before it traverses the network, but could put a lot of load on the agent if not kept simple.

[warkolm (Mark Walkom)]

Yep that's pretty much the reasoning behind the separation!

[mehdi-lamrani]

Ok, makes sense.
Thanks for the clarification