my logstash json filter seems be throwing errors on certain messages and I'm nut sure how to debug it. Is there a way to figure what messages or what is source of the message that threw this exception. I have multiple messages types from multiple sources coming into the pipeline.
I don't think there is a straightforward way to debug this easily.
Jackson (which is the library Logstash uses for JSON decoding iirc) accepts UTF-8, UTF-16 and UTF-32 encoded messages. Anything else will break it.
Also, the JSON filter cannot expose more information by default ( see relevant logging line here).
That said, if you really feel like pinpointing the issue, I would probably do the following:
Patch the local copy of the JSON filter under /vendor/bundle/jruby/1.9/gems/logstash-filter-json-x.x/lib/logstash/filters, and change the above linked line to @logger.warn("Error parsing json", :source => @source, :raw => source, :exception => e, :host => event.get('host')). This should allow you to at least pinpoint the offending device(s).
Optionally make a tcpdump of the traffic reaching your Logstash instance from said source(s) to check it's actual message payload if you can't debug it on the source's end.
I realize this might be bothersome or not feasible, so if anyone knows of a cleaner way, feel free to correct me.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.