I get occasional failed inserts/indexes because of malformed data (as does everyone, I assume).
With the new version, I'm getting a lot of the "Can't merge a non-object mapping with an object mapping" error, in part because I had to get rid of a lot of my object mappings.
In order to solve the problem, I need to see which data is bad. But I can't do that within logstash because logstash only gives me an error telling me the insert fail, not the output of the data. What would be great is if logstash would give me this data when it fails.
Now I can look in Elastic, but there's some issues there.
I don't actually know which log setting that will show me the failed index inserts. I don't need a million debug/trace lines. Just failed inserts.
Why is that important? Because I'm running 60k EPS and upping the logging verbosity is going to blast my local hard drive fairly quickly. It's actually undesirable for me to look in ES, because I'm running 50 elastic nodes. So when I see a failed insert on one of my logstash nodes, I have to go rifling through 50 ES nodes? That doesn't make sense.
What is the best way to do this? Preferably, I'd love to do it in logstash, because it knows the insert failed. But upping to 'trace' or 'debug' level verbosity is going to require a lot of hand-holding in order to ensure my hard drives don't fill.