Logstash multiline codec do not read all lines

Elastic Stack Logstash
1

Hi I'm using multiline codec to concatenate lines into one event, here's an example of it:

# Time: 2022-08-29T07:43:30.314166Z
# User@Host: root[root] @ 1.1.1.1. []  Id: 111111
# Query_time: 0.019870  Lock_time: 0.000000 Rows_sent: 0  Rows_examined: 0
SET timestamp=31554231;
set global slow_query_log_file="/logs/mysql-log";
# Time: 2022-08-29T12:34:30.419218Z
# User@Host: app[usr] @  [2.2.2.2]  Id: 2222222
# Query_time: 0.000037  Lock_time: 0.000000 Rows_sent: 0  Rows_examined: 0
use aaaaa;
SET timestamp=4141454;
SET autocommit=4;

this is my config:

input {
  file {
   path => "/home/user1/test.log"
   start_position => "beginning"
   sincedb_path => "/dev/null"
   codec => multiline {
   pattern => "^#%{SPACE}Time:%{SPACE}%{TIMESTAMP_ISO8601:time}"
   negate => true
   what => "previous"
  }
 }
}

Everything works, but the problem is that it does not read the second part of the log, namely everything that starts with

# Time: 2022-08-29T12:34:30.419218Z

however, if after "SET autocommit=4;" I add a similar timestamp "# Time: 2022-08-29....", then it will read everything up to "# Time: 2022-08-29....".
The question is how to make it read everything to the end?

2

You need to set the auto_flush_interval to get the last event.

From the documentation.

The accumulation of multiple lines will be converted to an event when either a matching new line is seen or there has been no new data appended for this many seconds. No default. If unset, no auto_flush.

Since you do not have anymore events, the last lines will not be converted to an event unless you set a timeout for the auto flush.

Try to add auto_flush_interval => 60 in your configuration, this will convert the accumulated lines into an event if you do not have any more matching lines for 60 seconds.

3

Thank you very much, I should read the documentation more carefully)

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.