Filter working on ubuntu but not on windows; mysterious "\r"

Elastic Stack Logstash
1

Hi, i have a working elastic stack linux setup for evaulation but I wanted to try out Windows 7 as server as well.. Because I have to backfill a lot of logs, I have to change the timestamp with the filter date to the real time from the logs. This works fine on ubuntu with the following filter:

input { stdin{}}

filter {
date {
locale =>"en"
match => ["message","YYYY-MM-dd;HH:mm:ss.SSS"]
timezone =>"Europe/Vienna"
target =>"@timestamp"
add_field => {"debug" => "timestampMatched"}
}
}

output { stdout {codec =>"rubydebug" }}

input:
2014-08-01;11:00:22.123
The output is:
{
"message" =>"2014-08-01;11:00:22.123",
"@version" =>"1",
"@timestamp" => "2014-08-01T09:00:22.123Z",
"host" =>"ABCDE",
"debug" =>"timestampMatched"
}

On Win7 (Win 7 Ultimate x64, jdk-8u74 x64, elasticSearch 2.2.0, logstash 2.2.2), i always get a dateparsefailure because of a mysterious \r... Who can help me solving this problem ?

c:\elastic\logstash\bin>logstash agent -f datetest.conf
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 1
Logstash startup completed
2016-03-13;15:33:22.123
?[33mFailed parsing date from field {:field=>"message", :value=>"2016-03-13;15:3
3:22.123\r", :exception=>"Invalid format: "2016-03-13;15:33:22.123\r" is malfo
rmed at "\r"", :config_parsers=>"YYYY-MM-dd;HH:mm:ss.SSS", :config_locale=>"en
", :level=>:warn}?[0m
{
"message" => "2016-03-13;15:33:22.123**\r**",
"@version" => "1",
"@timestamp" => "2016-03-13T22:22:23.889Z",
"host" => "WIN-F75UV5K32SV",
"tags" => [
[0] "_dateparsefailure"
]
}

1 Like
2

Did you copy the data over from linux, it could just be an encoding issue.
It'd try using stdin and then pasting to see if there is actually an issue.

3

The \r character is the carriage return, i.e. the first character in a Windows new line sequence. It appears Logstash regardless of what platform is runs on uses just \n as a line separator. If that's the case I'd say it's a bug. In the meantime I suspect you'll be able to use the mutate filter's strip option to remove the unwanted trailing whitespace character from the message field.

4

thank you @warkolm. i wrote the filters for ubuntu on my windows 10 office machine and copied the same file to the new win7 elastic stack installation, so the filter should work..

this example std filter works fine on my win7 setup (without the "\r problem"):

input { stdin { } }

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

so i think it is maybe a problem of the date filter + windows or a bug. like @magnusbaeck wrote. thank you for your answer! I will try to workaround with the mutate strip filter.

5

it worked to remove the trailing whitespace on windows with:

....
filter {
mutate {
strip => ["message"]

}
......

thanks again @magnusbaeck!