LogStash encoding Issue from Filebeat IIS Access Logs 7.4.0 Stack

[Nicholas_Penning]

I have tried many ways to get this warning message to go away in my LogStash pipeline but am failing to do so:

[2019-10-15T17:42:15,175][WARN ][logstash.filters.urldecode] Received an event that has a different character encoding than you configured. {:text=>"action=savesΓÇ¥, 1, ΓÇ£runMode=1&siteMode=1&siteHel
p=\xB1\xBE\xCD\xF8╒╛\xD2\xF2\xB3\xCC\xD0\xF2\xC9\xFD\xBC\xB6\xB9╪▒\xD5\xD6\xD0&SwitchComments=1", :expected_charset=>"UTF-8"}

I have tried this in the Filebeat Config:

• type: log
  encoding: utf-8

With this in the in LogStash inside of the input/beats:
codec => plain {
charset => "UTF-8"
}

With ingesting many web logs, we are bound to get some special characters of sorts. How can these special characters be properly dealt with?

[Badger]

@Nicholas_Penning

p=\xB1\xBE

I do not believe a UTF-8 string can contain two consecutive characters that both start with hex B. You have an input that expects UTF-8, and you are looking at solutions to tell it that the string is UTF-8. The problem is that the string is not UTF-8.

I suggest you Google something like "find string encoding" and recommend you also read this.

[Nicholas_Penning]

Thanks for the quick response, I will dig deeper on this. I was secretly hoping the the Filebeat encoding setting would do some magical work to encoding the incoming data appropriately. Maybe it's worth looking at the source and seeing if I can force an encoding when the logs get written to IIS instead. If any others have gone down this path, please let me know!

Thanks Badger!

[Badger]

logstash will never guess the encoding. You have to tell it.

Adding a front-end (a codec?) to some inputs that could guess the encoding is an interesting idea, but my gut response is that that would be a catastrophe for too many folks.

[TechGeekNZ (Andrew)]

Hey Nicholas,

Check your logging configuration in IIS. I wonder if you're logging is currently set to ANSI instead of UTF-8

Hope this helps

[Nicholas_Penning]

Very good suggestion, I will check this! Thank you!