Logstash - Charset 1252 - Problems with Conversion

Elastic Stack Logstash
1

Hi everyone

I have been struggling with Logstash to be able to apply the windows charset CP1252 in a UDP input listener. I'm running on a windows server 2012 with Logstash 5.2.2.

The charset setup seems does not make effect and after I sent the data I see the character like that:
Sent data : [é ó ção]
Result in Elastic : [\xE9 \xF3 \xE7\xE3o]

The UDP setup follows below.

input
{
udp
{
port => 5140
codec => plain
{
charset => "CP1252"
}
type => "log4net"
}
}

The output setup is an ElasticSearch as follows

output
{
stdout{
codec => rubydebug
}
if[type] =="log4net"{
elasticsearch{
hosts => ["localhost:9200"]
index => "log-%{+YYYY.MM.dd}"
}
}
}

I did a try sending direct to ElasticSearch without Logstash and it worked.

Any tips to solve it is very welcome

Thanks & Regards

2

After some more tests, it seems that the problem should be happen when grok apply the filter before it goes to output.
Has anyone experienced this issue?

Thanks

3

I don't see any grok in your config

4

Sorry, the filter is:

filter
{
if [type] == "log4net"
{
grok
{
remove_field => message
match => { message => "(?m)%{TIMESTAMP_ISO8601:sourceTimestamp} %{DATA:userName} %{WORD:machineName} %{DATA:loggerName}: %{DATA:threadId} %{LOGLEVEL:level} %{DATA:systemname} %{WORD:environment} %{WORD:site} %{GREEDYDATA:tempMessage}" }
}
if !("_grokparsefailure" in [tags])
{
mutate {
replace => [ "message" , "%{tempMessage}"]
}
}
mutate {
remove_field => [ "tempMessage" ]
remove_field => [ "tempHost" ]
}
}
}

5

I really don't understand your question, sorry.
What you mean

6

When I send a given message to logstash, lets say "[é ó ção]" it seems that after the filter process the is performed by a grok codec to transform the message, the initial value that was "[é ó ção]" is wrongly converted to this the characteres "\xE9 \xF3 \xE7\xE3o". I'm not understanding what's the reason of it's behavior . If I remove the filter step, the message arrives in the right way to the output since I can see it in Kibana.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.