Filebeat combines mulitple lines into one

Elastic Stack Beats
1

For some reason filebeat combines multiple lines in my log into a single event. Also for some reason filebeat is not shipping the metadata for these messages and I'm not sure if it's related.

sample source lines in a file:

2018-11-14T17:36:02.582113+00:00 localhost dropbear[22618]: Child connection from 192.168.1.1:35362
2018-11-14T17:36:02.584137+00:00 localhost dropbear[22618]: Exit before auth: Exited normally
2018-11-14T17:36:21.101821+00:00 localhost python: DIGEST-MD5 common mech free
2018-11-14T17:36:22.468729+00:00 localhost httpd: DIGEST-MD5 common mech free
2018-11-14T17:36:58.357544+00:00 localhost python: DIGEST-MD5 common mech free

filebeat config:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /app/log/auth.log

logstash config:

input {
beats {
port => 5044
codec => line
}
}

filter {
mutate {
add_field => { "test_field1" => "testing_field" }
}
mutate {
add_field => { "test_field2" => "%{@metadata}" }
}
}
output {
file {
path => "/var/logstash/nonparsed/output"
codec => rubydebug
}
}

file output:

{
"@timestamp" => 2018-11-14T17:38:35.305Z,
"test_field1" => "testing_field",
"message" => "2018-11-14T17:36:02.582113+00:00 localhost dropbear[22618]: Child connection from 192.168.1.1:353622018-11-14T17:36:02.584137+00:00 localhost dropbear[22618]: Exit before auth: Exited normally2018-11-14T17:36:21.101821+00:00 localhost python: DIGEST-MD5 common mech free2018-11-14T17:36:22.468729+00:00 localhost httpd: DIGEST-MD5 common mech free2018-11-14T17:36:58.357544+00:00 localhost python: DIGEST-MD5 common mech free",
"test_field2" => "{}",
"@version" => "1"
}

3

Can you post the complete filebeat configuration please?

Why did you configure the line codec with the beats input? One should not configure a codec in the beats input in Logstash.

4

That was it! Removed the codec option and things are normal again.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.