I recently updated to Logstash 7.7 from 7.6, after this update the following is occurring:
A typical log line looks like this:
2019-10-02T04:00:09.254Z - ^[[32minfo^[[39m: Activity:content-finish 35692
When processing from Filebeat, the filter fails with a "_grokparsefailure" and the resulting message line looks like this in stdout:
"message" => "2019-10-02T04:00:09.254Z - \e[32minfo\e[39m: Activity:content-finish 35692"
The difference is the LOGLEVEL ANSI Color codes, which seem to change from:
^[[32minfo^[[39m
to
\e[32minfo\e[39m
This was working correctly, before the update.
Here is my config:
input {
beats {
port => "5044"
}
}
filter {
if [message] =~ /^\d+/ {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}-%{SPACE}(\^\[\[\d+m)%{LOGLEVEL:level}(\^\[\[\d+m):%{SPACE}%{GREEDYDATA:message}" }
overwrite => { "message" => "%{message}" }
remove_field => [ "host", "agent" ]
}
grok {
match => { "[log][file][path]" => "/%{WORD}/%{WORD}/%{WORD}/%{GREEDYDATA:id}/%{NUMBER:uploadepoch}/%{GREEDYDATA:logname}.log*" }
}
date {
match => [ "timestamp", "ISO8601" ]
remove_field => [ "timestamp" ]
}
date {
match => [ "uploadepoch", "UNIX_MS" ]
target => "uploadDate"
remove_field => [ "uploadepoch" ]
}
ruby {
code => 'event.set( "logUploadYear", event.get("uploadDate").time.strftime("%Y"))'
remove_field => [ "uploadDate" ]
}
if "beats_input_codec_plain_applied" in [tags] {
mutate {
remove_tag => [ "beats_input_codec_plain_applied" ]
}
}
} else {
drop{ }
}
}
output {
if "_grokparsefailure" in [tags] {
stdout { codec => rubydebug }
}
}
Any Ideas?
I should note, I have tried changing the regex to represent what is coming through in stdout:
(\\e\[+\d+m)%{LOGLEVEL:level}(\\e\[+\d+m)
But this doesn't resolve the issue...