Logstash multiline codec ignore last event / line

[metju312]

Hello,

Logstash multiline codec ignore my last event (line) until send next packege of logs.

In extremely case - if log have only one line it looks like there is no event at all.

My logstash.conf:

input {
    }
	http {
        port => "5001"
        codec => multiline {
            pattern => "^\[%{TIMESTAMP_ISO8601}\]"
            negate => true
            what => previous
            auto_flush_interval => 15
        }
    }
}

filter{
    grok {
        match => { "message" => "(?m)\[%{TIMESTAMP_ISO8601:timestamp}\]\s\<%{LOGLEVEL:log-level}\>\s\[%{WORD:component}\]\s%{GREEDYDATA:log-message}" 
    }
}

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		index => "%{+YYYY-MM-dd}"
	}
}

Moreover solution with auto_flush_interval don't work.

For example:
input using Postman:

[2017-07-11 22:32:12.345] [KCU] Component initializing
Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
[2017-07-11 22:32:16.345] [KCU] Return with status 1

output:

[2017-07-11 22:32:12.345] [KCU] Component initializing
Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)

I need this last line.

Question:
Am I doing something wrong or there are problems with multiline codec?

[guyboertje]

The multiline codec auto-flush will not work for the http input at the moment. The http input needs to be made auto-flush aware - this is because auto-flushing operates out-of-band meaning that it needs access to the queue to add the event into.

Why are you sending two events in one http payload?

Using the plain (default) codec and one event per http push you should see a message field with the request body as a single string with newline characters in it.

If you must sent multiple events in one request you can try removing the multiline and adding the split filter with a terminator of "\n[". This unfortunately will remove the [ character from the message line of the second, third etc events. You can handle this with different grok patterns though or if you are using the message line as is then use mutate inside a regex conditional block to replace the message field with "[%{[message]}" to add the [ back in.