Logstash TCP Input Codec to set utf-16 is not working

[aliaramli]

Hi,

I wanted to set the input tcp plugin to have codec charset utf-16, because one of the logs has Chinese characters.

[2019-01-30T09:23:25,617][WARN ][logstash.codecs.line ] Received an event that has a different character encoding than you configured. {:text=>"<13>Jan 30 09:23:21 whatever.com 856\\t52\\t-\\t-\\t{\\\"publis\\\":\\\"ww\\\",\\\"ho\\\":\\\"w.com\\\",\\\"pub\\\":\\\"52\\\",\\\"date\\\":\\\"2019-01-30 09:23:21\\\",\\\"request_method\\\":\\\"GET\\\",\\\"content_length\\\":\\\"-\\\",\\\"content_type\\\":\\\"text/html\\\",\\\"cache_control\\\":\\\"-\\\",\\\"upst\\\":\\\"1164:80\\\",\\\"sc\\\":\\\"ht.232\\\",\\\"doci\\\":\\\"/aur/̸Ц\\xB2\\xBB\\xB7\\xE7\\xC9\\xFA\\\",\\\"upus\\\":\\\"200\\\",\\\"ed\\\":\\\"1717\\\",\\\"cac\\\":\\\"EXP\\\",\\\"op\\\":\\\"85\\\"}", :expected_charset=>"UTF-8"}

Unfortunately, my setup doesnt work.... the same error still showing

input {
  tcp {
    port => 15140
    type => "default"
    codec => plain {
      charset => "UTF-16"
    }   
  }
}

Please help.

[danhermann]

@aliaramli, are you sure that error message is the result of events you're receiving through the TCP input because your config shows the TCP input configured with the plain codec and the warning message above is from the line codec.

[aliaramli]

I ve changed to codec line, same issue still persist. Hmm if its not coming from tcp event, where could it come from? As i only set to accept input from tcp

  input {
      tcp {
        port => 15140
        type => "default"
        codec => line {
         charset => "UTF-16"
         }   
       }
      }

[danhermann]

I'd suggest gathering more information by running with a higher log level (use --log.level debug on the command line or its equivalent) and posting several of the logging lines before and after the error you're encountering.

[aliaramli]

Thank you so much, after running the debug mode, i noticed that its coming from another TCP input on different port.....which was defined in another separate config file.....
Thanks! managed to set the charset to utf-16 already!