Ingest Pipeline for custom logs

Elastic Stack Elastic Agent
1

Hi,
I am ingesting the standards syslogs using the elastic agent "custom logs" integration. I am trying to parse the logs before indexing and for that I am using ingest pipeline. I have two processors for that: Grok and remove in a sequential manner. When I test the pipeline in ingest pipeline it works but when I use in the custom log integration in the section custom configurations:
pipeline:example-ingest-pipeline and save and deploy the integration it throws error like this

image
image987×299 16.6 KB

Does anybody have idea on this?

2

Did you try putting a space after the :
pipeline: example-ingest-pipeline

3

@stephenb yes, I missed the space after pipeline. Error goes away but still pipeline is not integrated. I will update things how it goes. Thank you for the immediate response.

4

Can you show the screenshot of where you put that in.

You're using the custom logs integration?

That should work. I used that all the time.

Did you put any error handling in to see if it's failing in the ingest pipeline?

5

@stephenb Yes, it works. I just had to redo the integration and ingest pipeline. It wasn't working probably working because I was editing the the integration and pipeline. Thank you, now it works like a charm.

6

Hi @stephenb, now I could parse data and everything looks good. however, there is still one caveat - I can't run the KQL queries in Discover as it shows like this:

query
query1015×462 33.6 KB

My KQL queries run perfectly fine when I am ingesting data without the ingest pipeline via custom logs integration

7

What is the KQL Query .. what does the document Look Like

The KQL is probably not returning any results / no match

What does your documents look like and what exact KQL... need exact Examples

The ingest pipeline is creating different fields look at the documents / mappings ... more details otherwise we are just guessing

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.