Write data that have failed in Logstash filter transforms to file

My logstash config has a filter performing several transformations. But I see many records failing to get processed because of incorrect format/encoding in source data.
I want Logstash to write these failed inputs to a file, which I can further use to validate data loss and/or improve source data and logstash transforms.

How can I do this currently?

if you set log.level to debug or trace, Logstash write more information about failures to log file. Is it not enough for you?

But using debug I can't identify the ones that have failed. I want to view only the failed messages.

This post help you?

You need to share your logstash configuration, show what filters you are using.

Some parser filters, like grok, json, kv and csv will add a tag when it fails, other filters do that as well.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.