I have a need to add an ingest pipeline to the elastic-cloud-logs-9 data stream in a cloud-hosted environment running 9.2. It’s not complicated logic, but it’s a little complicated extracting a field from message with 2 different formats. I couldn’t get this to work, even though the simulate works, the template was updated and the index rolled over.
Then it gets stranger.
I added a SET to add a tag. just to show the pipeline was entered. Some of the events are tagged, the ones I want aren’t. Thinking my script was an error, I removed that (script) processor. So I have a pipeline with no logic, just adds a tag. Still some are tagged, some aren’t.
It looks only event.dataset == agent.log events go thru the pipeline, events that I want are from elasticsearch.server. There are others that aren’t tagged as well, elasticsearch.gc, elasticsearch.audit and kibana.log. I think these agent.log events come from one host, the other events are from other hosts, I think ALL are the cloud hosts.
I have a case open with Elastic, but I just wondered if anyone here has any ideas.
Thanks