For some reason filebeat combines multiple lines in my log into a single event. Also for some reason filebeat is not shipping the metadata for these messages and I'm not sure if it's related.
sample source lines in a file:
2018-11-14T17:36:02.582113+00:00 localhost dropbear[22618]: Child connection from 192.168.1.1:35362
2018-11-14T17:36:02.584137+00:00 localhost dropbear[22618]: Exit before auth: Exited normally
2018-11-14T17:36:21.101821+00:00 localhost python: DIGEST-MD5 common mech free
2018-11-14T17:36:22.468729+00:00 localhost httpd: DIGEST-MD5 common mech free
2018-11-14T17:36:58.357544+00:00 localhost python: DIGEST-MD5 common mech free
filebeat config:
filebeat.prospectors: - type: log enabled: true paths: - /app/log/auth.log
logstash config:
input {
beats {
port => 5044
codec => line
}
}filter {
mutate {
add_field => { "test_field1" => "testing_field" }
}
mutate {
add_field => { "test_field2" => "%{@metadata}" }
}
}
output {
file {
path => "/var/logstash/nonparsed/output"
codec => rubydebug
}
}
file output:
{
"@timestamp" => 2018-11-14T17:38:35.305Z,
"test_field1" => "testing_field",
"message" => "2018-11-14T17:36:02.582113+00:00 localhost dropbear[22618]: Child connection from 192.168.1.1:353622018-11-14T17:36:02.584137+00:00 localhost dropbear[22618]: Exit before auth: Exited normally2018-11-14T17:36:21.101821+00:00 localhost python: DIGEST-MD5 common mech free2018-11-14T17:36:22.468729+00:00 localhost httpd: DIGEST-MD5 common mech free2018-11-14T17:36:58.357544+00:00 localhost python: DIGEST-MD5 common mech free",
"test_field2" => "{}",
"@version" => "1"
}