How to found problematic messages?

[Vitaly_il (Vitaly)]

There are many warnings in my logstash logfile about messages with too many fields:

"reason"=>"Limit of total fields [1000] in index [logstash-type-YYYY.MM] has been exceeded"

How to see problematic messages? The only method I know about it's add "file" output filter to the Logstash config and look for a long messages. Running Logstash with 'debug' level seems me overkill.
I'm curious is there more elegant method exist.

Thank you,
Vitaly

[magnusbaeck (Magnus Bäck)]

This is ES complaining about the total number of fields in the index, so it's not necessarily a problem with any single message. If the problem is a single event with >1000 fields rather than, say, 100 events each having 10 unique fields you can use a ruby filter to count the number of fields in the event and e.g. tag the event so you have the opportunity to dump the event to another output.