Convert Field Data to UTF8

[MaikLinnemann (Maik)]

Dear Forum,

i am new to the ELK and i would like to convert data in a field called "subject" to UTF8. The data comes pushed from logstash to elasticsearch and visualized with kibana. My problem is that the data comes in an ugly encoding and i would like to have a way to convert it to UTF8. Could anyone give me a good start how i could solve my issue. Thanks in advance,

Regards,

[Stacey_Gammon]

What encoding is your data in when you index it into Elasticsearch? Elasticsearch and Kibana will support UTF-8. Can you convert it prior to indexing into Elasticsearch?

[MaikLinnemann (Maik)]

Hi Stacey,

i push the data via logstash like:

"subject" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },

Strictly speaking it is a subject line from postfix which i parse with grok patterns. I dont know a pretty way to convert it prior pushing as postfix logging cant convert it while logging. I thought that there might be a way to convert it from or with kibana (?).

As you can see, sometimes the subject is broken becaus of german characters.

I tried to convert it just for a test from the shell with a perl oneliner: cat test.txt | perl -MEncode -ne 'print encode("utf8",decode("MIME-Header",$_))' which beautifies the subject as expected.

[MaikLinnemann (Maik)]

Is similar converting with ELK possible? Could anyone give me a hint or a starting point to do so?