How to catch an error in logstash

Hello,

my logstash json filter seems be throwing errors on certain messages and I'm nut sure how to debug it. Is there a way to figure what messages or what is source of the message that threw this exception. I have multiple messages types from multiple sources coming into the pipeline.

[2018-10-22T06:36:16,228][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"�,\u001A~��\u0005X<���EIx]\u0014��3���E,����&��l�YT\u0018g����\u0018;�b�מ�y1�dY�\u0015�Yt�f�\u0005Xܘ߄Ι+J�,*~����\u0018\v���7��/ƒ��b�0Ϣ�7c,����\u0004��+J�,:~�\"��2��dw�%�7c.�?m�A\u0002rm~�����?m\u001E�|yn������1~�.��3�=�Gl^cMk��7/�ߌ�\u0000�������\u0015%a\u0016\u0015�yq�f�\u0005X����;\u0002{~��0Ϣ�7c,�b���#��\u0015%�uQ��\u0017�o�X����o\u0016%a\u0016\u0015�yq�f�\u0005X<��fQ\u0012�G+~����\u0018;�b�מ��i,ɲ\u0018+̳���\u0018\v��9����ܰ.ƒ�=:V�g���1\u0016q�o\u0002��\u0014%�uq�)�Iu�u�X���w����X\u0012f��Ic0�沟���\u0004��\u0016�M��g\u0002���f���+�Y��\u001F�t��v�R�\u0005����}\u0003\u001DK��\u008AS�:N)�\u0002,n;%��i,\t?\u007F(N��8�\u0018\v��93�~!�xG�;�Xa�E�)�X�����.��\u0004-*̳�8�\u0018\v��\u007F����S�)�", :exception=>#<LogStash::Json::ParserError: Invalid UTF-8 start byte 0xbf`

I don't think there is a straightforward way to debug this easily.
Jackson (which is the library Logstash uses for JSON decoding iirc) accepts UTF-8, UTF-16 and UTF-32 encoded messages. Anything else will break it.

Also, the JSON filter cannot expose more information by default ( see relevant logging line here).

That said, if you really feel like pinpointing the issue, I would probably do the following:

  • Patch the local copy of the JSON filter under /vendor/bundle/jruby/1.9/gems/logstash-filter-json-x.x/lib/logstash/filters, and change the above linked line to @logger.warn("Error parsing json", :source => @source, :raw => source, :exception => e, :host => event.get('host')). This should allow you to at least pinpoint the offending device(s).
  • Optionally make a tcpdump of the traffic reaching your Logstash instance from said source(s) to check it's actual message payload if you can't debug it on the source's end.

I realize this might be bothersome or not feasible, so if anyone knows of a cleaner way, feel free to correct me.

Thanks paz, it's better that nothing. I'll try it out.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.