Question about Multiline codec in combining the lines

Elastic Stack Logstash
1

I have a fake data file:
ST 1
EN 1
ST 2
EN 2
ST 3
EN 3

My config file is
input {
stdin {}
file {
# type => "tomcat"
path => "C:\1LOGS\XXX\data\fakedata4.log"
start_position => beginning
codec => multiline {
pattern => "^SR"
negate => "true"
what => "previous"
}
}
}
output {
stdout{codec => rubydebug}
}

but then the output only shows the first two events
Logstash startup completed
{
"@timestamp" => "2016-02-22T06:54:55.137Z",
"message" => "SR 1\r\nEN 1\r",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "C:\1LOGS\XXX\data\fakedata4.log",
"host" => "XXXX"
}
{
"@timestamp" => "2016-02-22T06:54:55.139Z",
"message" => "SR 2\r\nEN 2\r",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "C:\1LOGS\XXX\data\fakedata4.log",
"host" => "XXXX"
}

Q: why did the multi-line codec plugin seem to stop before SR 3?

Thanks!

2

I have a fake data file:
ST 1
EN 1
ST 2
EN 2
ST 3
EN 3

Judging by the rest of your post I'm going to assume you meant "SR" instead of "ST".

Q: why did the multi-line codec plugin seem to stop before SR 3?

Because it's still waiting for the next non-SR line. It can't flush the currently built-up lines until it sees an SR lines that signals that a new sequence of lines begins.

1 Like
3

Hi,

New to ELK stack, multiline filter works for me but i saw it is not thread safe as well as deprecated. So is there an alternate way to overcome this problem of not flushing the last line from the log file? I am currently using logstash-2.3.2 with 'file' input type to analyse the logs files. Please suggest.

Thank you!

4

The multiline codec doesn't work?

5

Yes using multiline codec but it is not flushing the last line of the log file. When I shut down the logstash, then it processes the last line.

6

I moved over to filebeat and this problem went away because filebeat with the multiline setting is reading every line.