Hi all,
my problem is the following: After upgrading "some" major versions, my logstash config doesn't work as expected anymore. I configured on several different containers and it should process the logs like before. On some hosts it doesn't even process the logs, on others only from the first file input. Was there any change, which could have broken my config? Only thing I changed, was the json_lines delimiter syntax.
Graylog version didn't change and I did the upgrade, because I refactored my logconfig from longer time ago.
Didn't touch anything else than my own logstash-forward.conf, thought acout configuring pipelines.yml, because I have up to 4 different patterns of logs in my hosts.
input {
file {
path => "/opt/java-app/logs/catalina.out"
sincedb_path => "/opt/java-applogstash/sincedb/opt_java-app_logs_catalina.out"
tags => ['streams-java-apps','timeformat_java']
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => "true"
what => "previous"
max_lines => 50000
}
}
if "timeformat_java" in [tags] {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp}"
}
}
grok {
match => {
"message" => "%{LOGLEVEL:loglevel}"
}
}
grok {
match => {
"message" => "%{JAVAEX:exception}"
}
}
date {
# 2015-06-26 09:45:14,439
match => ["timestamp","ISO8601","YYYY-MM-dd HH:mm:ss,SSS","YYYY-MM-dd HH:mm:ss.SSS","YYYY-MM-dd HH:mm:ss"]
}
kv { prefix => "param" }
}
ruby {
code => "event.set('timestamp', '%10.3f' % event.get('@timestamp').to_f)"
}
mutate {
convert => {"timestamp" => "float"}
}
mutate {
remove_field => ["message","@timestamp","@version"]
}
}
output {
tcp {
host => "graylog-host"
port => "12201"
codec => json_lines {
delimiter => 0
}
}
}