Logstash - Json Parse Failure and Encoding Problem

nevincansel · January 4, 2022, 1:38pm

Hello,

I have a pipeline with tcp input in logstash and pipeline is working without a problem but logs are broken. These are error logs in logstash.

13:34:43.868 [nioEventLoopGroup-2-2] WARN  logstash.codecs.jsonlines - Received an event that has a different character encoding than you configured. {:text=>"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5", :expected_charset=>"UTF-8"}
13:34:43.868 [nioEventLoopGroup-2-2] WARN  logstash.codecs.jsonlines - JSON parse error, original data now in message field {:message=>"Unrecognized token '_': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5\"; line: 1, column: 2]", :exception=>LogStash::Json::ParserError, :data=>"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5"

This is logstash pipeline.

  devopsdashboard.conf: |
    input {
      tcp {
        port => 9500
        codec => "json_lines"
      }
    }
    filter {
      mutate { 
        remove_field => ["host", "port"]
        }
    }
    output {
      elasticsearch {
        index => "devopsdashboard-%{+YYYY.MM.dd}"
        hosts => [ "elasticsearch-master:9200" ]
      }
    }

I am sending logs from nlog with encoding utf-8 to logstash. How can I fix the error?

Regards

nevincansel · January 10, 2022, 1:50pm

Any ideas?

leandrojmp · January 10, 2022, 2:38pm

This happens to every log or just some of them?

If you remove the codec option from your input, what do you have? This seems to be an issue with the source, not Logstash, as it is giving you warning in the input for the wrong encoding.

How are you sending the logs? Can you share any configuration or code with the part where you configure the codec and where you send it? Maybe this can give a hint of the issue.

nevincansel · January 11, 2022, 7:44am

Hi Leandro,

It is same for every log.

I removed codec and filter part and deployed logstash again. Here are new logs:


07:40:43.662 [nioEventLoopGroup-2-1] WARN  logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"R\\xECJB7Ҁ\\x9B\\u000F\\t2\\x8A.b\\xECّ\\xA6\\u0015D\\x9A\\xFBQ\\x91F\\u0002M\\xC2{", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN  logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\u0012\\x88(r{{ڮ\\xB0\\xCB\\u0003E\\xAA&i\\xBB\\xBB\\u0014\\xE1\\xD1H\\xF3\\u000EkJG\\xBB\\x91*\\xB6\\xF8iFN\\xFCik\\xFD\\xAEfx\\x9A\\u0001\\u00168\\\"\\u0011\\x96\\u0013\\u0004T}\\xA6?\\x80\\xE7\\xFB\\x9A\\xA1)\\u0003\\x8D\\xC5\\u0000\\x80\\xEC\\xDA5\\x87\\x9E\\xBD6_\\x84t\\xA1\\u0010\\xA4-\\xFA\\u0016\\x85*\\u0002\\xE5drY\\xC10)P\\xC7\\u0002e\\xD6\\b\\xEER\\xA2\\u007Fg\\x98ʃ\\u0012`Sf0\\u009E\\x88\\u0003\\u0000a\\xC1\\xA5@\\xF3\\u0001\\x97\\xC6\\u0084R͛\\b\\u0013\\xAA\\b\\xD9&\\x94\\x9C_\\xA9\\xA9\\x86f\\u0013\\xAA `\\u0016S\\x96C,\\x88ēJ\\xB9\\x95\\u001A\\xF7\\e\\x98r\\x8ENm\\xF4q\\u001C\\\"U\\xB0w\\xED$\\xB0\\xA9\\xA0\\xE6U\\xCE\\u0013>\\xD5\\u0013\\x9B1+\\u000F\\xEABE\\xD6\\u001EDr\\xBB\\xA7T~\\xE6A\\xCB\\u007F\\xA4\\u001E\\f\\xB9\\xDAl+3\\xB4\\x84\\xF4\\xF4\\xC1j\\xF9!\\xB6\\xF0\\u001Cq\\xC0%8\\xC6)\\a.s]\\xA1\\xB1Z3@\\r\\xD3*\\\"2'\\xCA\\u0004\\\\\\xC2\\u007F)\\u001A\\xFE\\xFC\\xE9c\\x8E\\x8B1\\xE3\\xBD\\xC9\\v\\xEAM\\x86\\x83P,\\xF7-nG\\xC6\\u0003\\x9D\\xFF,z\\x89'榌]d\\x84\\xED\\xA6\\u0011\\xF0\\xF6\\xF8+\\xF2.\\xF2eg\\xEE)q\\xEB\\xEC\\xCC\\xDD\\t̥\\xD59\\xE6\\xCC\\u0003g\\u0004\\xA4\\\"\\xE1\\xFB\\x82\\xA4:\\u0013\\x85", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN  logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"L1\\xA1n\\xE4t\\xDEr\\xC5ڿ&_\\xBB)\\xBF\\xFB\\xBB\\xEB]\\xCD\\xEA\\u001D\\xEE\\xBE\\xFB\\xBB\\e\\xE6\\xE3\\xC1\\xBB\\xBF\\xFB=\\xF3\\xB1]\\xBD\\u001E\\xBC\\xFB\\xBB;\\xEB\\xE1\\x82\\xFA(B8\\xD4\\xE0L\\x8A\\xC0\\xB1\\xD4\\xDDM\\xA3\\xFE+\\xF5J\\xA2+\\xE2:a|q\\x87r\\b\\xB2usD\\x94)\\xEF\\u0012\\u0005\\x9A\\xAE'=:\\xEA7\\xF4\\x89Qzz\\xA4\\x96:Xk\\x9F\\xF4\\xF6ƂH\\xD6d ȣ\\xCA\\u0018\\xC2&\\u000E8\\x91(B\\xEE;\\x97\\xE9\\x80S\\xBF\\xE5\\u0003\\x98\\xA0\\xA0\\xFE\\x9F\\x90~\\u00069ԅ\\x88\\xEC\\xE6\\xE3?\\x8D\\xE4\\u0014\\xA8\\xC4Ŵh`\\xB4#a*y|\\u001CSp\\xE1\\xE6\\xBF\\u0019\\xE6Eo[\\xE4\\xAA\\xCD8\\xDC \\en\\u0010/\\x83 s\\xD1\\xEE{\\u0003K\\xB1t\\xB5\\xD6\\xD2\\u0006\\xFA\\xE3n\\xBD}\\xD8?\\xEC\\f\\u0014\\xA3>\\xD0\\u001F;\\xF3~zMW;\\x8A\\xC1\\xEE[\\u001D#\\xC2-\\v0\\xBD\\u0012%ۤ9\\x9F\\xFF\\x93G{\\\"\\x8F\\xBDL\\xBD\\xDBҎ\\xDDׄ\\xEDih\\xAF\\xBD\\u0019\\x8B\\x86qiVƊ\\xC9X\\xBD\\u001F\\xDB:%\\u0017\\x9F\\xB3\\x85Q\\u0014\\xC0\\f", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN  logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\xC1L\\xAE0:\\x84\\xA0\\x9C|kF&\\x9A/\\x97o\\u0005\\xD1\\xD4\\u0001\\xAE\\x92Qx\\u0017*\\xFF\\x87a^\\xF4\\\\*\\xA8\\xA4W\\xC3\\xCC\\u0012\\xA2r\\xF0\\xF6\\x97_\\b\\xE9rN\\x80*\\u0012d", :expected_charset=>"UTF-8"}

I send logs with Nlog file in json format. 30123 is NodePort of Logstash routes to 9500.

<?xml version="1.0" encoding="utf-8" ?>
<!-- XSD manual extracted from package NLog.Schema: https://www.nuget.org/packages/NLog.Schema -->
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xsi:schemaLocation="NLog NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">

  <!-- the targets to write to -->
  <targets >
    <!-- write logs to file  -->
    <target name="logstashInternal" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true">
      <layout type='JsonLayout'>
        <attribute name="LogLine" layout="${callsite-linenumber}"  />
        <attribute name='TS' layout='${date:format=yyyy-MM-ddTHH\:mm\:ss.fff}' />
        <attribute name="G" layout="${event-context:item=trackId}" />
        <attribute name="UserId" layout="${event-context:item=userId}" />
        <attribute name='LV' layout='${level:upperCase=true}'/>
        <attribute name="MN" layout="${machinename}" />
        <attribute name="LG" layout="${logger}"/>
        <attribute name="MN" layout="${callsite}"  />
        <attribute name="LogLine" layout="${callsite-linenumber}"  />
        <attribute name='message' layout='${message}' />
      </layout>
    </target>
    <target name="logstashException" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true">
      <layout type='JsonLayout'>
        <attribute name='TS' layout='${date:format=yyyy-MM-ddTHH\:mm\:ss.fff}' />
        <attribute name="G" layout="${event-context:item=trackId}" />
        <attribute name="UserId" layout="${event-context:item=userId}" />
        <attribute name='LV' layout='${level:upperCase=true}'/>
        <attribute name="MN" layout="${machinename}" />
        <attribute name="LG" layout="${logger}"/>
        <attribute name="MN" layout="${callsite}"  />
        <attribute name="LogLine" layout="${callsite-linenumber}"  />
        <attribute name='message' layout='${message}' />
        <attribute name='exception' layout='${exception:maxInnerExceptionLevel=3:format=ToString}' />
      </layout>
    </target>
  </targets>

  <!-- rules to map from logger name to target -->
  <rules>
    <!--<logger name="*" minlevel="Trace" writeTo="InternalLogger,console" />-->
    <logger name="ExceptionLogger" minlevel="Error" writeTo="logstashException" />
    <logger name="InternalLogger" minlevel="Trace" writeTo="logstashInternal" />
  </rules>

</nlog>

As you can see, xml is using utf-8 as encoding but logstash gives error.

leandrojmp · January 11, 2022, 12:42pm

For some reason your log is not being sent as utf-8.

I do not know C#/dotnet, but google some similar questions it seems that you need to specify the encoding in the target line, as the one you have in the top of your xml file is the encoding for this xml file, not the encoding that will be used to send logs.

Try changing that targets to something like this:

<target name="logstashInternal" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8">

and

<target name="logstashException" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8">

nevincansel · January 12, 2022, 7:44am

I changed encoding as you said alsı writeBom to nlog file but nothing has changed.

 <target name="logstashInternal" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8" writeBom="true">
<target name="logstashException" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8" writeBom="true">

07:41:49.783 [nioEventLoopGroup-2-1] WARN  logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\xC5݁\\xDAi\\xAA\\xBD擫\\u0019L(^\\xF4\\xDE\\xD8\\xC0ם\\xEDQ\\x8B\\u0011\\xAE\\xA6\\x9F\\xCD\\u0004\\xA3\\x8D\\xFB\\xA3\\xB6o\\x81\\xC3V߷\\eyc6(\\xA5\\x88\\xFA\\u0014t\\xFE#\\xEB\\u0012\\x8F\\x8F(#\\u001Ea\\x9E\\xBE^\\xABz\\xA5\\u000E\\u001E<\\xA9<yKy\\xE7w\\xD6;ڠ;<~\\xE7w6̓\\xFE;\\xBF\\xF3\\xB6yBt\\xE3\\xFE;\\xBF\\xB3\\xB5P\\xCD\\u0015\\u0014\\u0000@\\x9E2A\\xD4s\\x8B\\u0013m\\xB2\\eTB\\x81\\xE6E\\xFA;\\x91\\xB1rɩ\\u0013\\xA1N\\xAB\\xC1\\xB84\\u001FQާ) @\\xFCU\\u0001*B\\x81\\x8C\\xC1\\u0002;\\xB8\\xE6M\\xCA\\f\\x8C\\xD3\\u0014v\\xFE3\\xF6J\\xA2\\u0005q\\u001D\\x97O\\xE6\\xF3\\x86\\xFBP\\u0013\\xCB<\\xB8\\xD2:\\u0006\\x880\\b\\xAB\\xE0AT\\u00804vbI\\xE1\\u0005\\xCA\\u0003\\xF9x%\\xE3\\xF1aё\\x87\\x8E\\xDA\\xD5\\u0003{\\u007F\\xDD\\xFD\\xE6Q\\xB8\\u001D*p$Mg\\xD3\\xF3\\xCC\\xD0$\\xE4\\xB1\\u0005\\xD6@\\u0012\\xB7x\\xBAm\\xC3\\xCD\\u007F1̍7ϑ\\xA2\\f\\u0001\\xB9R@P\\xA0\\xA8p\\xC1\\xB6\\x93\\xFD\\x81:\\xD0\\e\\xA5\\xA6\\xD6\\xD7O:k\\xADao\\xD8\\xEE\\xAB\\xC6Z_?\\xB1f\\xFA\\xF4\\x92\\xDEh\\xAB\\u0006\\xFB`\\xD06\\u0002ܲ \\xE2\\u0017ҿ8\\xE9J\\xE9\\xFC\\x9F\\a\\x9B~\\xFB^\\xA6\\xDEij\\xE7\\xF45!23>\\xAD*\\u001Fv\\x83\\u0013\\xEAZ~\\u000E\\x8A@\\xAA\\xE56#WA\\b<p\\x9A\\x92\\x83\\xA6\\xC9i\\x83oi\\xB6z\\xD9sk\\u0015\\xF1x\\xA8P\\xFF\\xCD07\\x9E\\x89\\u0005\\x95\\xFC\\xE2U(\\xCA> ,\\u0002Q\\xE1\\xA1\\xC2\\aN\\x82N8", :expected_charset=>"UTF-8"}
07:41:49.784 [nioEventLoopGroup-2-1] WARN  logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\x9Ei\\x9D\\xA6\\xD9\\xEB\\xAF\\u001D\\x9B\\xE6\\xA0?\\xE8\\xA9\\xDDRS\\u001Dh=\\xB5s\\xA2u\\xF5FK\\xEB\\xAD\\xF9\\xFE\\x9Bm\\xF4\\xAF\\xDDrP\\\"M\\xE02\\xE0X\\b\\xA4`\\u0010\\x9Db9\\xD8\\xD5z}\\xB3\\xD3ь\\xB5a\\x97\\xBC\\x9Ew\\xE8i'z\\u007F\\xA0\\xF56\\x9B7\\x81\\x8CE\\u0000\\u0001\\xA2\\xFB\\xECm\\u0014\\x8D\\x93\\xEFZ\\u0011=w\\x85\\u001E\\u001F~s\\u0018\\xC8i\\xA6FD*\\x82C\\u0019\\\"82\\x85\\x9DQ\\x8CI\\xC5h\\x9D\\u0018@[ymtϟ\\u001A\\x98\\u0005\\xA6H\\u001F\\xA6\\xFD\\b=\\xB3\\xE5aQ3\\x98\\xC9S\\xBE\\xF9r\\x86a", :expected_charset=>"UTF-8"}

nevincansel · January 21, 2022, 11:13am

Do you have any other ideas?

leandrojmp · January 21, 2022, 1:26pm

Unfortunatelly no, I do not .net/C#.

The issue here is that your source is not sending logs with the right encoding even when configured to send using utf-8, it is not an issue with logstash, you will need to try to solve that first.

system · February 18, 2022, 1:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.