Bad charset encoding in field names (II)

[Thiago_Ruiz]

Hi!

I have exactly the same issue as described by Ondřej Bílka on this thread:

Bad charset encoding in field names

I have an issue with ElasticSearch after upgrade to 5.6. I am generating documents using logstash and dynamic fields using native Ruby extension in logstash: filter { if [attributes] =~ /[^\s\\]+/ { kv { source => "attributes" field_split => ";" value_split => ":" target => "attributes" } ruby { code => " event.get('attributes').each do |cusField| event.set(cusField[0], cusField[1].split(',')) end " } } In these dynamic fields, there are ke…

Since the discussion was closed I'm trying to reopen it again.

All my files are in UTF-8 and just the dinamic fields are wrong - I have one 'message' field with all the data and on that field the special characters are ok.

Thanks.

[Ondrej_Bilka (Ondřej Bílka)]

Hi, my only solution was to change the keys to web url-safe slugs...

[Thiago_Ruiz]

Sorry to hear that....

I'll try to solve this issue a little longer.

This was not a problem until I updated all ELK tools from 5.5.2 to 6.2.3

[Ondrej_Bilka (Ondřej Bílka)]

I wish you good luck and if you will figure anything out, please let me know..
On the other hand I think that "webalized" string keys are much more secured because of possible encoding issues in another parts of application.

[Thiago_Ruiz]

Yeah... i gave up and used this on my ugly field inside of a ruby filter:

event.set(strKey.tr( "ÀÁÂÃÄÅàáâãäåAaAaAaÇçCcCcCcCcÐðDdÐdÈÉÊËèéêëEeEeEeEeEeGgGgGgGgHhHhÌÍÎÏìíîïIiIiIiIiIiJjKk?LlLlLl??LlÑñNnNnNn???ÒÓÔÕÖØòóôõöøOoOoOoRrRrRrSsSsSsŠš?TtTtTtÙÚÛÜùúûüUuUuUuUuUuUuWwÝýÿYyŸZzZzŽž", "AAAAAAaaaaaaAaAaAaCcCcCcCcCcDdDdDdEEEEeeeeEeEeEeEeEeGgGgGgGgHhHhIIIIiiiiIiIiIiIiIiJjKkkLlLlLlLlLlNnNnNnNnnNnOOOOOOooooooOoOoOoRrRrRrSsSsSsSssTtTtTtUUUUuuuuUuUuUuUuUuUuWwYyyYyYZzZzZz"), strValue.to_f)

Don't know how is the performance of this, though I dont have a high volume of data going through this code..

[Kyo_Hwang]

There is still a problem when I use a docker image sebp/elk 6.2.3 or 6.2.4.