# Logstash - Json Parse Failure and Encoding Problem

**URL:** <https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447>\
**Category:** Logstash\
**Created:** [January 4, 2022, 1:38pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447 "2022-01-04T13:38:42Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nevincansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nevincansel/32/98534_2.png) [@nevincansel](https://discuss.elastic.co/u/nevincansel)\
**Post date:** [January 4, 2022, 1:38pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/1 "2022-01-04T13:38:42Z")

</div>

Hello,

I have a pipeline with tcp input in logstash and pipeline is working without a problem but logs are broken. These are error logs in logstash.

```auto
13:34:43.868 [nioEventLoopGroup-2-2] WARN logstash.codecs.jsonlines - Received an event that has a different character encoding than you configured. {:text=>"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5", :expected_charset=>"UTF-8"}
13:34:43.868 [nioEventLoopGroup-2-2] WARN logstash.codecs.jsonlines - JSON parse error, original data now in message field {:message=>"Unrecognized token '_': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5\"; line: 1, column: 2]", :exception=>LogStash::Json::ParserError, :data=>"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5"

```

This is logstash pipeline.

```auto
  devopsdashboard.conf: |
    input {
      tcp {
        port => 9500
        codec => "json_lines"
      }
    }
    filter {
      mutate { 
        remove_field => ["host", "port"]
        }
    }
    output {
      elasticsearch {
        index => "devopsdashboard-%{+YYYY.MM.dd}"
        hosts => ["elasticsearch-master:9200"]
      }
    }

```

I am sending logs from nlog with encoding utf-8 to logstash. How can I fix the error?

Regards

---

<div class="post-metadata">

**Author:** ![nevincansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nevincansel/32/98534_2.png) [@nevincansel](https://discuss.elastic.co/u/nevincansel)\
**Post date:** [January 10, 2022, 1:50pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/2 "2022-01-10T13:50:56Z")

</div>

Any ideas?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 10, 2022, 2:38pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/3 "2022-01-10T14:38:55Z")

</div>

This happens to every log or just some of them?

If you remove the `codec` option from your input, what do you have? This seems to be an issue with the source, not Logstash, as it is giving you warning in the input for the wrong encoding.

How are you sending the logs? Can you share any configuration or code with the part where you configure the codec and where you send it? Maybe this can give a hint of the issue.

---

<div class="post-metadata">

**Author:** ![nevincansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nevincansel/32/98534_2.png) [@nevincansel](https://discuss.elastic.co/u/nevincansel)\
**Post date:** [January 11, 2022, 7:44am UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/4 "2022-01-11T07:44:59Z")

</div>

Hi Leandro,

It is same for every log.

I removed codec and filter part and deployed logstash again. Here are new logs:

```auto

07:40:43.662 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"R\\xECJB7Ҁ\\x9B\\u000F\\t2\\x8A.b\\xECّ\\xA6\\u0015D\\x9A\\xFBQ\\x91F\\u0002M\\xC2{", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\u0012\\x88(r{{ڮ\\xB0\\xCB\\u0003E\\xAA&i\\xBB\\xBB\\u0014\\xE1\\xD1H\\xF3\\u000EkJG\\xBB\\x91*\\xB6\\xF8iFN\\xFCik\\xFD\\xAEfx\\x9A\\u0001\\u00168\\\"\\u0011\\x96\\u0013\\u0004T}\\xA6?\\x80\\xE7\\xFB\\x9A\\xA1)\\u0003\\x8D\\xC5\\u0000\\x80\\xEC\\xDA5\\x87\\x9E\\xBD6_\\x84t\\xA1\\u0010\\xA4-\\xFA\\u0016\\x85*\\u0002\\xE5drY\\xC10)P\\xC7\\u0002e\\xD6\\b\\xEER\\xA2\\u007Fg\\x98ʃ\\u0012`Sf0\\u009E\\x88\\u0003\\u0000a\\xC1\\xA5@\\xF3\\u0001\\x97\\xC6\\u0084R͛\\b\\u0013\\xAA\\b\\xD9&\\x94\\x9C_\\xA9\\xA9\\x86f\\u0013\\xAA `\\u0016S\\x96C,\\x88ēJ\\xB9\\x95\\u001A\\xF7\\e\\x98r\\x8ENm\\xF4q\\u001C\\\"U\\xB0w\\xED$\\xB0\\xA9\\xA0\\xE6U\\xCE\\u0013>\\xD5\\u0013\\x9B1+\\u000F\\xEABE\\xD6\\u001EDr\\xBB\\xA7T~\\xE6A\\xCB\\u007F\\xA4\\u001E\\f\\xB9\\xDAl+3\\xB4\\x84\\xF4\\xF4\\xC1j\\xF9!\\xB6\\xF0\\u001Cq\\xC0%8\\xC6)\\a.s]\\xA1\\xB1Z3@\\r\\xD3*\\\"2'\\xCA\\u0004\\\\\\xC2\\u007F)\\u001A\\xFE\\xFC\\xE9c\\x8E\\x8B1\\xE3\\xBD\\xC9\\v\\xEAM\\x86\\x83P,\\xF7-nG\\xC6\\u0003\\x9D\\xFF,z\\x89'榌]d\\x84\\xED\\xA6\\u0011\\xF0\\xF6\\xF8+\\xF2.\\xF2eg\\xEE)q\\xEB\\xEC\\xCC\\xDD\\t̥\\xD59\\xE6\\xCC\\u0003g\\u0004\\xA4\\\"\\xE1\\xFB\\x82\\xA4:\\u0013\\x85", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"L1\\xA1n\\xE4t\\xDEr\\xC5ڿ&_\\xBB)\\xBF\\xFB\\xBB\\xEB]\\xCD\\xEA\\u001D\\xEE\\xBE\\xFB\\xBB\\e\\xE6\\xE3\\xC1\\xBB\\xBF\\xFB=\\xF3\\xB1]\\xBD\\u001E\\xBC\\xFB\\xBB;\\xEB\\xE1\\x82\\xFA(B8\\xD4\\xE0L\\x8A\\xC0\\xB1\\xD4\\xDDM\\xA3\\xFE+\\xF5J\\xA2+\\xE2:a|q\\x87r\\b\\xB2usD\\x94)\\xEF\\u0012\\u0005\\x9A\\xAE'=:\\xEA7\\xF4\\x89Qzz\\xA4\\x96:Xk\\x9F\\xF4\\xF6ƂH\\xD6d ȣ\\xCA\\u0018\\xC2&\\u000E8\\x91(B\\xEE;\\x97\\xE9\\x80S\\xBF\\xE5\\u0003\\x98\\xA0\\xA0\\xFE\\x9F\\x90~\\u00069ԅ\\x88\\xEC\\xE6\\xE3?\\x8D\\xE4\\u0014\\xA8\\xC4Ŵh`\\xB4#a*y|\\u001CSp\\xE1\\xE6\\xBF\\u0019\\xE6Eo[\\xE4\\xAA\\xCD8\\xDC \\en\\u0010/\\x83 s\\xD1\\xEE{\\u0003K\\xB1t\\xB5\\xD6\\xD2\\u0006\\xFA\\xE3n\\xBD}\\xD8?\\xEC\\f\\u0014\\xA3>\\xD0\\u001F;\\xF3~zMW;\\x8A\\xC1\\xEE[\\u001D#\\xC2-\\v0\\xBD\\u0012%ۤ9\\x9F\\xFF\\x93G{\\\"\\x8F\\xBDL\\xBD\\xDBҎ\\xDDׄ\\xEDih\\xAF\\xBD\\u0019\\x8B\\x86qiVƊ\\xC9X\\xBD\\u001F\\xDB:%\\u0017\\x9F\\xB3\\x85Q\\u0014\\xC0\\f", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\xC1L\\xAE0:\\x84\\xA0\\x9C|kF&\\x9A/\\x97o\\u0005\\xD1\\xD4\\u0001\\xAE\\x92Qx\\u0017*\\xFF\\x87a^\\xF4\\\\*\\xA8\\xA4W\\xC3\\xCC\\u0012\\xA2r\\xF0\\xF6\\x97_\\b\\xE9rN\\x80*\\u0012d", :expected_charset=>"UTF-8"}

```

I send logs with Nlog file in json format. 30123 is NodePort of Logstash routes to 9500.

```auto
<?xml version="1.0" encoding="utf-8" ?>
<!-- XSD manual extracted from package NLog.Schema: https://www.nuget.org/packages/NLog.Schema -->
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xsi:schemaLocation="NLog NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">

  <!-- the targets to write to -->
  <targets >
    <!-- write logs to file -->
    <target name="logstashInternal" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true">
      <layout type='JsonLayout'>
        <attribute name="LogLine" layout="${callsite-linenumber}" />
        <attribute name='TS' layout='${date:format=yyyy-MM-ddTHH\:mm\:ss.fff}' />
        <attribute name="G" layout="${event-context:item=trackId}" />
        <attribute name="UserId" layout="${event-context:item=userId}" />
        <attribute name='LV' layout='${level:upperCase=true}'/>
        <attribute name="MN" layout="${machinename}" />
        <attribute name="LG" layout="${logger}"/>
        <attribute name="MN" layout="${callsite}" />
        <attribute name="LogLine" layout="${callsite-linenumber}" />
        <attribute name='message' layout='${message}' />
      </layout>
    </target>
    <target name="logstashException" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true">
      <layout type='JsonLayout'>
        <attribute name='TS' layout='${date:format=yyyy-MM-ddTHH\:mm\:ss.fff}' />
        <attribute name="G" layout="${event-context:item=trackId}" />
        <attribute name="UserId" layout="${event-context:item=userId}" />
        <attribute name='LV' layout='${level:upperCase=true}'/>
        <attribute name="MN" layout="${machinename}" />
        <attribute name="LG" layout="${logger}"/>
        <attribute name="MN" layout="${callsite}" />
        <attribute name="LogLine" layout="${callsite-linenumber}" />
        <attribute name='message' layout='${message}' />
        <attribute name='exception' layout='${exception:maxInnerExceptionLevel=3:format=ToString}' />
      </layout>
    </target>
  </targets>

  <!-- rules to map from logger name to target -->
  <rules>
    <!--<logger name="*" minlevel="Trace" writeTo="InternalLogger,console" />-->
    <logger name="ExceptionLogger" minlevel="Error" writeTo="logstashException" />
    <logger name="InternalLogger" minlevel="Trace" writeTo="logstashInternal" />
  </rules>

</nlog>

```

As you can see, xml is using utf-8 as encoding but logstash gives error.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 11, 2022, 12:42pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/5 "2022-01-11T12:42:19Z")

</div>

For some reason your log is not being sent as `utf-8`.

I do not know C#/dotnet, but google some similar questions it seems that you need to specify the encoding in the target line, as the one you have in the top of your `xml` file is the encoding for this `xml` file, not the encoding that will be used to send logs.

Try changing that targets to something like this:

```auto
<target name="logstashInternal" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8">

```

and

```auto
<target name="logstashException" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8">

```

---

<div class="post-metadata">

**Author:** ![nevincansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nevincansel/32/98534_2.png) [@nevincansel](https://discuss.elastic.co/u/nevincansel)\
**Post date:** [January 12, 2022, 7:44am UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/6 "2022-01-12T07:44:03Z")

</div>

I changed encoding as you said alsı writeBom to nlog file but nothing has changed.

```auto
 <target name="logstashInternal" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8" writeBom="true">
<target name="logstashException" xsi:type="Network" address="tcp://cluster_ip:30123" newLine="true" encoding="utf-8" writeBom="true">

```

```auto
07:41:49.783 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\xC5݁\\xDAi\\xAA\\xBD擫\\u0019L(^\\xF4\\xDE\\xD8\\xC0ם\\xEDQ\\x8B\\u0011\\xAE\\xA6\\x9F\\xCD\\u0004\\xA3\\x8D\\xFB\\xA3\\xB6o\\x81\\xC3V߷\\eyc6(\\xA5\\x88\\xFA\\u0014t\\xFE#\\xEB\\u0012\\x8F\\x8F(#\\u001Ea\\x9E\\xBE^\\xABz\\xA5\\u000E\\u001E<\\xA9<yKy\\xE7w\\xD6;ڠ;<~\\xE7w6̓\\xFE;\\xBF\\xF3\\xB6yBt\\xE3\\xFE;\\xBF\\xB3\\xB5P\\xCD\\u0015\\u0014\\u0000@\\x9E2A\\xD4s\\x8B\\u0013m\\xB2\\eTB\\x81\\xE6E\\xFA;\\x91\\xB1rɩ\\u0013\\xA1N\\xAB\\xC1\\xB84\\u001FQާ) @\\xFCU\\u0001*B\\x81\\x8C\\xC1\\u0002;\\xB8\\xE6M\\xCA\\f\\x8C\\xD3\\u0014v\\xFE3\\xF6J\\xA2\\u0005q\\u001D\\x97O\\xE6\\xF3\\x86\\xFBP\\u0013\\xCB<\\xB8\\xD2:\\u0006\\x880\\b\\xAB\\xE0AT\\u00804vbI\\xE1\\u0005\\xCA\\u0003\\xF9x%\\xE3\\xF1aё\\x87\\x8E\\xDA\\xD5\\u0003{\\u007F\\xDD\\xFD\\xE6Q\\xB8\\u001D*p$Mg\\xD3\\xF3\\xCC\\xD0$\\xE4\\xB1\\u0005\\xD6@\\u0012\\xB7x\\xBAm\\xC3\\xCD\\u007F1̍7ϑ\\xA2\\f\\u0001\\xB9R@P\\xA0\\xA8p\\xC1\\xB6\\x93\\xFD\\x81:\\xD0\\e\\xA5\\xA6\\xD6\\xD7O:k\\xADao\\xD8\\xEE\\xAB\\xC6Z_?\\xB1f\\xFA\\xF4\\x92\\xDEh\\xAB\\u0006\\xFB`\\xD06\\u0002ܲ \\xE2\\u0017ҿ8\\xE9J\\xE9\\xFC\\x9F\\a\\x9B~\\xFB^\\xA6\\xDEij\\xE7\\xF45!23>\\xAD*\\u001Fv\\x83\\u0013\\xEAZ~\\u000E\\x8A@\\xAA\\xE56#WA\\b<p\\x9A\\x92\\x83\\xA6\\xC9i\\x83oi\\xB6z\\xD9sk\\u0015\\xF1x\\xA8P\\xFF\\xCD07\\x9E\\x89\\u0005\\x95\\xFC\\xE2U(\\xCA> ,\\u0002Q\\xE1\\xA1\\xC2\\aN\\x82N8", :expected_charset=>"UTF-8"}
07:41:49.784 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\x9Ei\\x9D\\xA6\\xD9\\xEB\\xAF\\u001D\\x9B\\xE6\\xA0?\\xE8\\xA9\\xDDRS\\u001Dh=\\xB5s\\xA2u\\xF5FK\\xEB\\xAD\\xF9\\xFE\\x9Bm\\xF4\\xAF\\xDDrP\\\"M\\xE02\\xE0X\\b\\xA4`\\u0010\\x9Db9\\xD8\\xD5z}\\xB3\\xD3ь\\xB5a\\x97\\xBC\\x9Ew\\xE8i'z\\u007F\\xA0\\xF56\\x9B7\\x81\\x8CE\\u0000\\u0001\\xA2\\xFB\\xECm\\u0014\\x8D\\x93\\xEFZ\\u0011=w\\x85\\u001E\\u001F~s\\u0018\\xC8i\\xA6FD*\\x82C\\u0019\\\"82\\x85\\x9DQ\\x8CI\\xC5h\\x9D\\u0018@[ymtϟ\\u001A\\x98\\u0005\\xA6H\\u001F\\xA6\\xFD\\b=\\xB3\\xE5aQ3\\x98\\xC9S\\xBE\\xF9r\\x86a", :expected_charset=>"UTF-8"}

```

---

<div class="post-metadata">

**Author:** ![nevincansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nevincansel/32/98534_2.png) [@nevincansel](https://discuss.elastic.co/u/nevincansel)\
**Post date:** [January 21, 2022, 11:13am UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/7 "2022-01-21T11:13:52Z")

</div>

Do you have any other ideas?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 21, 2022, 1:26pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/8 "2022-01-21T13:26:32Z")

</div>

Unfortunatelly no, I do not .net/C#.

The issue here is that your source is not sending logs with the right encoding even when configured to send using utf-8, it is not an issue with logstash, you will need to try to solve that first.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 18, 2022, 1:27pm UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-and-encoding-problem/293447/9 "2022-02-18T13:27:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.