How to catch an error in logstash

[zahodi (Valentin)]

Hello,

my logstash json filter seems be throwing errors on certain messages and I'm nut sure how to debug it. Is there a way to figure what messages or what is source of the message that threw this exception. I have multiple messages types from multiple sources coming into the pipeline.

[2018-10-22T06:36:16,228][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"�,\u001A~��\u0005X<���EIx]\u0014��3���E,����&��l�YT\u0018g����\u0018;�b�מ�y1�dY�\u0015�Yt�f�\u0005Xܘ߄Ι+J�,*~����\u0018\v���7��/ƒ��b�0Ϣ�7c,����\u0004��+J�,:~�\"��2��dw�%�7c.�?m�A\u0002rm~�����?m\u001E�|yn������1~�.��3�=�Gl^cMk��7/�ߌ�\u0000�������\u0015%a\u0016\u0015�yq�f�\u0005X����;\u0002{~��0Ϣ�7c,�b���#��\u0015%�uQ��\u0017�o�X����o\u0016%a\u0016\u0015�yq�f�\u0005X<��fQ\u0012�G+~����\u0018;�b�מ��i,ɲ\u0018+̳���\u0018\v��9����ܰ.ƒ�=:V�g���1\u0016q�o\u0002��\u0014%�uq�)�Iu�u�X���w����X\u0012f��Ic0�沟���\u0004��\u0016�M��g\u0002���f���+�Y��\u001F�t��v�R�\u0005����}\u0003\u001DK��\u008AS�:N)�\u0002,n;%��i,\t?\u007F(N��8�\u0018\v��93�~!�xG�;�Xa�E�)�X�����.��\u0004-*̳�8�\u0018\v��\u007F����S�)�", :exception=>#<LogStash::Json::ParserError: Invalid UTF-8 start byte 0xbf`

[paz]

I don't think there is a straightforward way to debug this easily.
Jackson (which is the library Logstash uses for JSON decoding iirc) accepts UTF-8, UTF-16 and UTF-32 encoded messages. Anything else will break it.

Also, the JSON filter cannot expose more information by default ( see relevant logging line here).

That said, if you really feel like pinpointing the issue, I would probably do the following:

• Patch the local copy of the JSON filter under /vendor/bundle/jruby/1.9/gems/logstash-filter-json-x.x/lib/logstash/filters, and change the above linked line to @logger.warn("Error parsing json", :source => @source, :raw => source, :exception => e, :host => event.get('host')). This should allow you to at least pinpoint the offending device(s).
• Optionally make a tcpdump of the traffic reaching your Logstash instance from said source(s) to check it's actual message payload if you can't debug it on the source's end.

I realize this might be bothersome or not feasible, so if anyone knows of a cleaner way, feel free to correct me.

[zahodi (Valentin)]

Thanks paz, it's better that nothing. I'll try it out.