Ingest Pipeline for custom logs

I am ingesting the standards syslogs using the elastic agent "custom logs" integration. I am trying to parse the logs before indexing and for that I am using ingest pipeline. I have two processors for that: Grok and remove in a sequential manner. When I test the pipeline in ingest pipeline it works but when I use in the custom log integration in the section custom configurations:
pipeline:example-ingest-pipeline and save and deploy the integration it throws error like this

Does anybody have idea on this?

Did you try putting a space after the :
pipeline: example-ingest-pipeline

@stephenb yes, I missed the space after pipeline. Error goes away but still pipeline is not integrated. I will update things how it goes. Thank you for the immediate response.

Can you show the screenshot of where you put that in.

You're using the custom logs integration?

That should work. I used that all the time.

Did you put any error handling in to see if it's failing in the ingest pipeline?

@stephenb Yes, it works. I just had to redo the integration and ingest pipeline. It wasn't working probably working because I was editing the the integration and pipeline. Thank you, now it works like a charm.

Hi @stephenb, now I could parse data and everything looks good. however, there is still one caveat - I can't run the KQL queries in Discover as it shows like this:

My KQL queries run perfectly fine when I am ingesting data without the ingest pipeline via custom logs integration

What is the KQL Query .. what does the document Look Like

The KQL is probably not returning any results / no match

What does your documents look like and what exact KQL... need exact Examples

The ingest pipeline is creating different fields look at the documents / mappings ... more details otherwise we are just guessing

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.