Write data that have failed in Logstash filter transforms to file

[GitSpree23 (LogSpree)]

My logstash config has a filter performing several transformations. But I see many records failing to get processed because of incorrect format/encoding in source data.
I want Logstash to write these failed inputs to a file, which I can further use to validate data loss and/or improve source data and logstash transforms.

How can I do this currently?

[Tomo_M (Tomohiro Mitani)]

if you set log.level to debug or trace, Logstash write more information about failures to log file. Is it not enough for you?

[GitSpree23 (LogSpree)]

But using debug I can't identify the ones that have failed. I want to view only the failed messages.

[Tomo_M (Tomohiro Mitani)]

This post help you?

Logstash Handling Filter Errors

Errors are saved in an array named tags. For example Grok will add _grokparsefailure to that tag when something happens. Then you can do your error handling from that. if "_grokparsefailure" in [tags] { # do something } _grokparsefailure documentation _mutate_error The tag_on_failure option is what you are looking for and you can set it to what you want or use the defaults.

[leandrojmp (Leandro Pereira)]

You need to share your logstash configuration, show what filters you are using.

Some parser filters, like grok, json, kv and csv will add a tag when it fails, other filters do that as well.