I have a pipeline with tcp input in logstash and pipeline is working without a problem but logs are broken. These are error logs in logstash.
13:34:43.868 [nioEventLoopGroup-2-2] WARN logstash.codecs.jsonlines - Received an event that has a different character encoding than you configured. {:text=>"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5", :expected_charset=>"UTF-8"}
13:34:43.868 [nioEventLoopGroup-2-2] WARN logstash.codecs.jsonlines - JSON parse error, original data now in message field {:message=>"Unrecognized token '_': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5\"; line: 1, column: 2]", :exception=>LogStash::Json::ParserError, :data=>"_\\x92A\\x9E\\a\\u0094\\xBA\\xB7_\\u0004\\x8D\\u0019O(a)\\x92VЫ\\u001A\\xD7@K\\xFA\\xEF\\xD8[\\x85\\xD2@f`\\u0001\\x88\\u0005\\xC4a\\u001F0\\u0012\\aG\\x8D\\x8B:k\\xDBw\\xAB\\xE5\\xEDʮ\\xE7\\u0002q}\\xF5"
If you remove the codec option from your input, what do you have? This seems to be an issue with the source, not Logstash, as it is giving you warning in the input for the wrong encoding.
How are you sending the logs? Can you share any configuration or code with the part where you configure the codec and where you send it? Maybe this can give a hint of the issue.
I removed codec and filter part and deployed logstash again. Here are new logs:
07:40:43.662 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"R\\xECJB7Ҁ\\x9B\\u000F\\t2\\x8A.b\\xECّ\\xA6\\u0015D\\x9A\\xFBQ\\x91F\\u0002M\\xC2{", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\u0012\\x88(r{{ڮ\\xB0\\xCB\\u0003E\\xAA&i\\xBB\\xBB\\u0014\\xE1\\xD1H\\xF3\\u000EkJG\\xBB\\x91*\\xB6\\xF8iFN\\xFCik\\xFD\\xAEfx\\x9A\\u0001\\u00168\\\"\\u0011\\x96\\u0013\\u0004T}\\xA6?\\x80\\xE7\\xFB\\x9A\\xA1)\\u0003\\x8D\\xC5\\u0000\\x80\\xEC\\xDA5\\x87\\x9E\\xBD6_\\x84t\\xA1\\u0010\\xA4-\\xFA\\u0016\\x85*\\u0002\\xE5drY\\xC10)P\\xC7\\u0002e\\xD6\\b\\xEER\\xA2\\u007Fg\\x98ʃ\\u0012`Sf0\\u009E\\x88\\u0003\\u0000a\\xC1\\xA5@\\xF3\\u0001\\x97\\xC6\\u0084R͛\\b\\u0013\\xAA\\b\\xD9&\\x94\\x9C_\\xA9\\xA9\\x86f\\u0013\\xAA `\\u0016S\\x96C,\\x88ēJ\\xB9\\x95\\u001A\\xF7\\e\\x98r\\x8ENm\\xF4q\\u001C\\\"U\\xB0w\\xED$\\xB0\\xA9\\xA0\\xE6U\\xCE\\u0013>\\xD5\\u0013\\x9B1+\\u000F\\xEABE\\xD6\\u001EDr\\xBB\\xA7T~\\xE6A\\xCB\\u007F\\xA4\\u001E\\f\\xB9\\xDAl+3\\xB4\\x84\\xF4\\xF4\\xC1j\\xF9!\\xB6\\xF0\\u001Cq\\xC0%8\\xC6)\\a.s]\\xA1\\xB1Z3@\\r\\xD3*\\\"2'\\xCA\\u0004\\\\\\xC2\\u007F)\\u001A\\xFE\\xFC\\xE9c\\x8E\\x8B1\\xE3\\xBD\\xC9\\v\\xEAM\\x86\\x83P,\\xF7-nG\\xC6\\u0003\\x9D\\xFF,z\\x89'榌]d\\x84\\xED\\xA6\\u0011\\xF0\\xF6\\xF8+\\xF2.\\xF2eg\\xEE)q\\xEB\\xEC\\xCC\\xDD\\t̥\\xD59\\xE6\\xCC\\u0003g\\u0004\\xA4\\\"\\xE1\\xFB\\x82\\xA4:\\u0013\\x85", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"L1\\xA1n\\xE4t\\xDEr\\xC5ڿ&_\\xBB)\\xBF\\xFB\\xBB\\xEB]\\xCD\\xEA\\u001D\\xEE\\xBE\\xFB\\xBB\\e\\xE6\\xE3\\xC1\\xBB\\xBF\\xFB=\\xF3\\xB1]\\xBD\\u001E\\xBC\\xFB\\xBB;\\xEB\\xE1\\x82\\xFA(B8\\xD4\\xE0L\\x8A\\xC0\\xB1\\xD4\\xDDM\\xA3\\xFE+\\xF5J\\xA2+\\xE2:a|q\\x87r\\b\\xB2usD\\x94)\\xEF\\u0012\\u0005\\x9A\\xAE'=:\\xEA7\\xF4\\x89Qzz\\xA4\\x96:Xk\\x9F\\xF4\\xF6ƂH\\xD6d ȣ\\xCA\\u0018\\xC2&\\u000E8\\x91(B\\xEE;\\x97\\xE9\\x80S\\xBF\\xE5\\u0003\\x98\\xA0\\xA0\\xFE\\x9F\\x90~\\u00069ԅ\\x88\\xEC\\xE6\\xE3?\\x8D\\xE4\\u0014\\xA8\\xC4Ŵh`\\xB4#a*y|\\u001CSp\\xE1\\xE6\\xBF\\u0019\\xE6Eo[\\xE4\\xAA\\xCD8\\xDC \\en\\u0010/\\x83 s\\xD1\\xEE{\\u0003K\\xB1t\\xB5\\xD6\\xD2\\u0006\\xFA\\xE3n\\xBD}\\xD8?\\xEC\\f\\u0014\\xA3>\\xD0\\u001F;\\xF3~zMW;\\x8A\\xC1\\xEE[\\u001D#\\xC2-\\v0\\xBD\\u0012%ۤ9\\x9F\\xFF\\x93G{\\\"\\x8F\\xBDL\\xBD\\xDBҎ\\xDDׄ\\xEDih\\xAF\\xBD\\u0019\\x8B\\x86qiVƊ\\xC9X\\xBD\\u001F\\xDB:%\\u0017\\x9F\\xB3\\x85Q\\u0014\\xC0\\f", :expected_charset=>"UTF-8"}
07:40:43.663 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\xC1L\\xAE0:\\x84\\xA0\\x9C|kF&\\x9A/\\x97o\\u0005\\xD1\\xD4\\u0001\\xAE\\x92Qx\\u0017*\\xFF\\x87a^\\xF4\\\\*\\xA8\\xA4W\\xC3\\xCC\\u0012\\xA2r\\xF0\\xF6\\x97_\\b\\xE9rN\\x80*\\u0012d", :expected_charset=>"UTF-8"}
I send logs with Nlog file in json format. 30123 is NodePort of Logstash routes to 9500.
For some reason your log is not being sent as utf-8.
I do not know C#/dotnet, but google some similar questions it seems that you need to specify the encoding in the target line, as the one you have in the top of your xml file is the encoding for this xml file, not the encoding that will be used to send logs.
07:41:49.783 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\xC5݁\\xDAi\\xAA\\xBD擫\\u0019L(^\\xF4\\xDE\\xD8\\xC0ם\\xEDQ\\x8B\\u0011\\xAE\\xA6\\x9F\\xCD\\u0004\\xA3\\x8D\\xFB\\xA3\\xB6o\\x81\\xC3V߷\\eyc6(\\xA5\\x88\\xFA\\u0014t\\xFE#\\xEB\\u0012\\x8F\\x8F(#\\u001Ea\\x9E\\xBE^\\xABz\\xA5\\u000E\\u001E<\\xA9<yKy\\xE7w\\xD6;ڠ;<~\\xE7w6̓\\xFE;\\xBF\\xF3\\xB6yBt\\xE3\\xFE;\\xBF\\xB3\\xB5P\\xCD\\u0015\\u0014\\u0000@\\x9E2A\\xD4s\\x8B\\u0013m\\xB2\\eTB\\x81\\xE6E\\xFA;\\x91\\xB1rɩ\\u0013\\xA1N\\xAB\\xC1\\xB84\\u001FQާ) @\\xFCU\\u0001*B\\x81\\x8C\\xC1\\u0002;\\xB8\\xE6M\\xCA\\f\\x8C\\xD3\\u0014v\\xFE3\\xF6J\\xA2\\u0005q\\u001D\\x97O\\xE6\\xF3\\x86\\xFBP\\u0013\\xCB<\\xB8\\xD2:\\u0006\\x880\\b\\xAB\\xE0AT\\u00804vbI\\xE1\\u0005\\xCA\\u0003\\xF9x%\\xE3\\xF1aё\\x87\\x8E\\xDA\\xD5\\u0003{\\u007F\\xDD\\xFD\\xE6Q\\xB8\\u001D*p$Mg\\xD3\\xF3\\xCC\\xD0$\\xE4\\xB1\\u0005\\xD6@\\u0012\\xB7x\\xBAm\\xC3\\xCD\\u007F1̍7ϑ\\xA2\\f\\u0001\\xB9R@P\\xA0\\xA8p\\xC1\\xB6\\x93\\xFD\\x81:\\xD0\\e\\xA5\\xA6\\xD6\\xD7O:k\\xADao\\xD8\\xEE\\xAB\\xC6Z_?\\xB1f\\xFA\\xF4\\x92\\xDEh\\xAB\\u0006\\xFB`\\xD06\\u0002ܲ \\xE2\\u0017ҿ8\\xE9J\\xE9\\xFC\\x9F\\a\\x9B~\\xFB^\\xA6\\xDEij\\xE7\\xF45!23>\\xAD*\\u001Fv\\x83\\u0013\\xEAZ~\\u000E\\x8A@\\xAA\\xE56#WA\\b<p\\x9A\\x92\\x83\\xA6\\xC9i\\x83oi\\xB6z\\xD9sk\\u0015\\xF1x\\xA8P\\xFF\\xCD07\\x9E\\x89\\u0005\\x95\\xFC\\xE2U(\\xCA> ,\\u0002Q\\xE1\\xA1\\xC2\\aN\\x82N8", :expected_charset=>"UTF-8"}
07:41:49.784 [nioEventLoopGroup-2-1] WARN logstash.codecs.line - Received an event that has a different character encoding than you configured. {:text=>"\\x9Ei\\x9D\\xA6\\xD9\\xEB\\xAF\\u001D\\x9B\\xE6\\xA0?\\xE8\\xA9\\xDDRS\\u001Dh=\\xB5s\\xA2u\\xF5FK\\xEB\\xAD\\xF9\\xFE\\x9Bm\\xF4\\xAF\\xDDrP\\\"M\\xE02\\xE0X\\b\\xA4`\\u0010\\x9Db9\\xD8\\xD5z}\\xB3\\xD3ь\\xB5a\\x97\\xBC\\x9Ew\\xE8i'z\\u007F\\xA0\\xF56\\x9B7\\x81\\x8CE\\u0000\\u0001\\xA2\\xFB\\xECm\\u0014\\x8D\\x93\\xEFZ\\u0011=w\\x85\\u001E\\u001F~s\\u0018\\xC8i\\xA6FD*\\x82C\\u0019\\\"82\\x85\\x9DQ\\x8CI\\xC5h\\x9D\\u0018@[ymtϟ\\u001A\\x98\\u0005\\xA6H\\u001F\\xA6\\xFD\\b=\\xB3\\xE5aQ3\\x98\\xC9S\\xBE\\xF9r\\x86a", :expected_charset=>"UTF-8"}
The issue here is that your source is not sending logs with the right encoding even when configured to send using utf-8, it is not an issue with logstash, you will need to try to solve that first.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.