Logstash multiline pattern : continuous newline


#1

This is my DB logs(CUBRID Slow query log).

'16-09-26 14:26:31.564 (35) execute_all srv_h_id 14 DELETE FROM MSG_RCV_CC WHERE RCV_ID = ?
16-09-26 14:26:31.564 (35) bind 1 : VARCHAR (9)11282366
16-09-26 14:26:33.987 (35) execute_all 0 tuple 0 time 2.423'

16-09-26 15:47:19.344 (24) execute_all srv_h_id 3 INSERT INTO MSG_RCV_TEMP_QUEUE (RCV_ID, SERVER_SIGN,CREATE_DATE) VALUES (?, ?, CURRENT_TIMESTAMP)
16-09-26 15:47:19.344 (24) bind 1 : BIGINT 11292925
16-09-26 15:47:19.344 (24) bind 2 : VARCHAR (16)tcwhpsns-91a901
16-09-26 15:47:19.346 (24) execute_all error:-670 tuple 1 time 0.001, EID = 10

16-11-03 11:45:54.036 (24407) execute_all srv_h_id 3 INSERT INTO MSG_RCV_TEMP_QUEUE (RCV_ID, SERVER_SIGN,CREATE_DATE) VALUES (?, ?, CURRENT_TIMESTAMP)
16-11-03 11:45:54.036 (24407) bind 1 : BIGINT 11304157
16-11-03 11:45:54.036 (24407) bind 2 : VARCHAR (16)tcwhpsns-91a901
16-11-03 11:45:54.037 (24407) execute_all error:-670 tuple 1 time 0.001, EID = 214

I use multiline in my config file. I want to divide this log by empty line.
But I couldn't split what I wanted. Because Each Line has similar patterns and I can't match empty line as regex.
I need your help.


(Magnus Bäck) #2

I can't match empty line as regex.

Why not?

multitline {
  pattern => "^$"
  what => "previous"
  negate => true
}

#3

Thanks for reply .

It doesn't work properly. multline never end.
I use mulitline like this. Is it any problem? I use logstash 6.1.1 version

input {
stdin {
codec => multiline {
pattern => "^$"
what => "previous"
negate => true
}
}
}
filter {
grok {
match => { "message" => "%{DATE:date} %{TIME:time} %{DATA:id} %{GREEDYDATA:query}" }
}
if [query] =~ "error:" {
dissect { mapping => { "query" => "execute_all error:%{error} tuple %{tuple} time %{execute_time}, EID = %{EID}" } }
}
else if [query] =~ "tuple" {
dissect { mapping => { "query" => "execute_all 0 tuple %{tuple} time %{execute_time}" } }
}
else if [query] =~ "bind" {
dissect { mapping => { "query" => "bind 1 : %{bind}" } }
}
if "_grokparsefailure" in [tags] { drop{} }
if ([execute_time]) { mutate { convert => { "execute_time" => "float" }}}
}
output {
stdout { codec => "rubydebug" }
}


(Magnus Bäck) #4

What does your test input look like? Have you looked into the codec's auto_flush_interval option?


#5

my test input looks like this.

16-09-26 14:26:31.564 (35) execute_all srv_h_id 14 DELETE FROM MSG_RCV_CC WHERE RCV_ID = ?
16-09-26 14:26:31.564 (35) bind 1 : VARCHAR (9)11282366
16-09-26 14:26:33.987 (35) execute_all 0 tuple 0 time 2.423

16-09-26 15:47:19.344 (24) execute_all srv_h_id 3 INSERT INTO MSG_RCV_TEMP_QUEUE (RCV_ID, SERVER_SIGN,CREATE_DATE) VALUES (?, ?, CURRENT_TIMESTAMP)
16-09-26 15:47:19.344 (24) bind 1 : BIGINT 11292925
16-09-26 15:47:19.344 (24) bind 2 : VARCHAR (16)tcwhpsns-91a901
16-09-26 15:47:19.346 (24) execute_all error:-670 tuple 1 time 0.001, EID = 10

16-11-03 11:45:54.036 (24407) execute_all srv_h_id 3 INSERT INTO MSG_RCV_TEMP_QUEUE (RCV_ID, SERVER_SIGN,CREATE_DATE) VALUES (?, ?, CURRENT_TIMESTAMP)
16-11-03 11:45:54.036 (24407) bind 1 : BIGINT 11304157
16-11-03 11:45:54.036 (24407) bind 2 : VARCHAR (16)tcwhpsns-91a901
16-11-03 11:45:54.037 (24407) execute_all error:-670 tuple 1 time 0.001, EID = 214

16-11-03 11:47:48.806 (24407) execute_all srv_h_id 3 INSERT INTO MSG_RCV_TEMP_QUEUE (RCV_ID, SERVER_SIGN,CREATE_DATE) VALUES (?, ?, CURRENT_TIMESTAMP)
16-11-03 11:47:48.806 (24407) bind 1 : BIGINT 11304187
16-11-03 11:47:48.806 (24407) bind 2 : VARCHAR (16)tcwhpsns-91a901
16-11-03 11:47:48.808 (24407) execute_all error:-670 tuple 1 time 0.001, EID = 215

16-11-03 11:51:34.351 (24407) execute_all srv_h_id 3 INSERT INTO MSG_RCV_TEMP_QUEUE (RCV_ID, SERVER_SIGN,CREATE_DATE) VALUES (?, ?, CURRENT_TIMESTAMP)
16-11-03 11:51:34.351 (24407) bind 1 : BIGINT 11304189
16-11-03 11:51:34.351 (24407) bind 2 : VARCHAR (16)tcwhpsns-91a901
16-11-03 11:51:34.353 (24407) execute_all error:-670 tuple 1 time 0.001, EID = 216

I've never looked into auto_flush_interval option....


(Magnus Bäck) #6

How do you know your drop filter isn't dropping all events? If I delete all filters things work just fine.


#7

I've just tried what you said.
conf file that I've tried looks like this.

input {
stdin {
codec => multiline {
pattern => "^$"
what => "previous"
negate => true
}
}
}
filter {
}
output {
stdout { codec => "rubydebug" }
}

And It doesn't work either.

I think multiline couldn't catch empty line with this pattern.


(Magnus Bäck) #8

Well, I copied and pasted your configuration (without the filters) and it worked with Logstash 2.4.0. If I can't reproduce the problem I can't help you.


#9

Okay I understand. thanks for all your replies.
It may be version problem or something.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.