LogStash encoding Issue from Filebeat IIS Access Logs 7.4.0 Stack

I have tried many ways to get this warning message to go away in my LogStash pipeline but am failing to do so:

[2019-10-15T17:42:15,175][WARN ][logstash.filters.urldecode] Received an event that has a different character encoding than you configured. {:text=>"action=savesΓÇ¥, 1, ΓÇ£runMode=1&siteMode=1&siteHel
p=\xB1\xBE\xCD\xF8╒╛\xD2\xF2\xB3\xCC\xD0\xF2\xC9\xFD\xBC\xB6\xB9╪▒\xD5\xD6\xD0&SwitchComments=1", :expected_charset=>"UTF-8"}

I have tried this in the Filebeat Config:

  • type: log
    encoding: utf-8

With this in the in LogStash inside of the input/beats:
codec => plain {
charset => "UTF-8"

With ingesting many web logs, we are bound to get some special characters of sorts. How can these special characters be properly dealt with?

I do not believe a UTF-8 string can contain two consecutive characters that both start with hex B. You have an input that expects UTF-8, and you are looking at solutions to tell it that the string is UTF-8. The problem is that the string is not UTF-8.

I suggest you Google something like "find string encoding" and recommend you also read this.

Thanks for the quick response, I will dig deeper on this. I was secretly hoping the the Filebeat encoding setting would do some magical work to encoding the incoming data appropriately. Maybe it's worth looking at the source and seeing if I can force an encoding when the logs get written to IIS instead. If any others have gone down this path, please let me know!

Thanks Badger!

logstash will never guess the encoding. You have to tell it.

Adding a front-end (a codec?) to some inputs that could guess the encoding is an interesting idea, but my gut response is that that would be a catastrophe for too many folks.

Hey Nicholas,

Check your logging configuration in IIS. I wonder if you're logging is currently set to ANSI instead of UTF-8


Hope this helps

Very good suggestion, I will check this! Thank you!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.