Grok filter cannot parse log lines that are AFTER a multiline event

Elastic Stack Logstash
1

Hello everyone,

I have a really annoying problem.
As I mention in the title if I have an single-line event after a multiline event Logstash just freezes and cannot parse the rest of the file.
However, If that same single line is before the multiline event then Logstash has no problem parsing it.

My conf file is:

input {
    file {
        sincedb_path => "/dev/null/" # For debugging only!
        path => "/home/pantheo/Documents/ELK-playground/etenderingError-test.log"
        start_position => "beginning"
        codec => multiline {
            patterns_dir => ["./patterns"] 
            pattern => "^%{LINE_BEGIN}"
            negate => true
            what => "previous"
        }
    }
}

filter {
    if "multiline" in [tags]{
    grok {  
    patterns_dir => ["./patterns"]
    match => {"message" => "%{LINE_BEGIN:start-of-line} %{CUSTOM_TIMESTAMP:the-time} %{THREAD_AGENT:agent} %{USER:user-ID} %{REQUEST_ID:request-ID} %{STATUS:status} %{DATA:message-body}\n((?m)%{GREEDYDATA:stack-trace})?"}
    }
    }
    else{
    grok {
    patterns_dir => ["./patterns"]
    match => {"message" => "%{LINE_BEGIN:start-of-line} %{CUSTOM_TIMESTAMP:the-time} %{THREAD_AGENT:agent} %{USER:user-ID} %{REQUEST_ID:request-ID} %{STATUS:status} %{GREEDYDATA:message-body}"}
    }}
                 

    date {
         match => [ "timestamp", "dd/MM/yyyy HH:mm:ss.SSS" ] 
    }    
}

output {
    stdout { codec => rubydebug } 
}

and the file that works has the following contents:

>>> 12/11/2015 15:50:08.054 [http-0.0.0.0-8080-3] [Anonymous@10.0.0.1, 10.0.0.1] [reqId:2611] ERROR - .integrations.ws.WSClient - Error occurred while calling [SystemLogon] WS
>>> 11/11/2015 15:50:08.054 [http-0.0.0.0-8080-3] [Anonymous@10.0.0.1, 10.0.0.1] [reqId:2611] ERROR - s.ws.WSClient - Error occurred while calling [SystemLogon]
java.lang.RuntimeException: java.lang.RuntimeException: Error occurred while making a SOAP call to [https://quality-]
    at EIDWSClient.makeSOAPcall(EIDWSClient.java:59)
    at call(SystemLogonWSClient.java:74)

Logstash is able to parse and grok (and everything) the above file without problems

However if I transfer the first line (>>> 12/11/2015...) after this line "at call(SystemLogonWSClient.java:74)"

Logstash just hangs, unable to parse it.

2

It "hangs" because it is expecting that to be a multiline event and only ever sees the start of that event, not the end, so it cannot move to the next event.

3

Thanks for your reply.

I was thinking the same thing but can not figure out why it cannot move to the next event.

Shouldn't it just break out of the multiline event when the pattern is not satisfied?

4

No, that'd defeat the purpose.

5

So, I have to explicitly tell it when to break out?

6

That is correct.

7

Thank you for your help

8

Just a last update for anyone that might have a similar problem.

My problem is (exactly )what driskell refers to in his comment here:

Files are currently treated as streams of lines that are constantly
being written. So if we hit a line without EOL (end of line, \n) we
wait.