Logstash 6.0 http input not accepting utf-8

[Conall_Prendergast (Cónall Prendergast)]

Hi,

I have a logstash pipeline with a http input plugin. I want to use a json codec but this fails when the input contains the character "é".

This fails with the error:

[ERROR] 2017-11-15 11:32:24.663 [Ruby-0-Thread-22: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:61] json - JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): was expecting double-quote to start field name

This is because the input then no longer looks like json, and goes from

{\"key\": \"Ascii text with e\"}

to

{\\\"key\\\": \\\"Non ascii text with \\xE9\\\"}

The difference between the two messages above are the fact that the latter contains the character é.

how can I force the http input plugin to accept utf-8?

I already have the collowing line in my config, but it does not work:

codec => json { charset => "UTF-8" }

Thanks

LS 6.0 rc2

[guyboertje]

You are hitting this bug. https://github.com/elastic/logstash/issues/4560

Have you tried to remove the charset setting?

e.g.

codec => json { }

The charset setting is more useful when the incoming character set is not UTF-8 already.

[Conall_Prendergast (Cónall Prendergast)]

Thanks.
I added

codec => json {charset => "ISO-8859-1"}

to match the other end, even though I wanted it to use UTF-8. But it seems to work now.
Thanks

[guyboertje]

Yes. the charset setting is a from not a to value. The unwritten to charset is always UTF-8.