Filebeat stops if there is any failure in ingest pipeline

Elastic Stack Beats
1

Hi ,

I'm using filebeat -->ingest pipeline -->elasticsearch , and i'm using grok processor with set of log patterns under ingest pipeline . If there any log event which is not matching with grok processor patterns then filebeat halts and it will not ship further log events. then i have to manually restart filebeat to ship next log events

note: i'm using on_failure functionality for my pipeline but still the same issue.

Kindly help

2

There is something weird here. Especially if you are using on_failure which does not generate an HTTP error status.

I moved your question to #beats:filebeat. Could you provide may be more details? How you pipeline has been created, logs from filebeat?...

3

Can you also share your filebeat config, filebeat version and log output? Filebeat should not halt on error.

4

Thanks !! its working fine now. I was using 3 filebeat.prospector with log input_type and each was referring to different ingest pipeline. And there was one ingest pipeline which was missing "on_failure"
hence the issue.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.