Multiline not reading all lines?

Elastic Stack Logstash
1

I have some logs all starting with a timestamp, javaclass, log level and then a message, some of them multiline.
I'm just starting with a couple single-line events first. I have two of them in a file and I have the following config:

input{
    file{
        path => "/path/to/log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        codec => multiline{
            # not starting with a timestamp should be merged with the previous line
            pattern => "^%{TIMESTAMP_ISO8601}"
            negate => true
            what => "previous"
        }
    }
}
filter{
    grok{
        match => { "message" => "\A%{TIMESTAMP_ISO8601}%{SPACE}%{JAVACLASS} \[%{LOGLEVEL}] %{GREEDYDATA}" }
    }
}
output{
    stdout{codec => rubydebug}
}

When I run it prints out the first event but not the second. I checked the files and both have a hard return at the end of their line. I've even added a third, empty line to see if it would force the display of the second event but it didn't. What am I doing wrong?

1 Like
2

The problem is that Logstash can't know until it gets the next physical line that begins with a timestamp that the current logical line is finished and should be flushed. I think https://github.com/logstash-plugins/logstash-codec-multiline/issues/11 is the best issue to follow for tracking this limitation.

3

Thanks Magnus, makes sense.