Strang things happened when logstash grok the log message


#1

I use filebeat-1.2.2 to monitor the mysql-slow.log.
when I add two slow query records in it,

and the logstash only give me one :

:disappointed_relieved: I do it again,adding two more records into the mysql-slow.log:


hehe....:joy: just like the first time,logstash only give me one result:

:imp: I tried the third time:

and aha!!! :joy: this I got two results.I don't know why....

could you tell me why ? :joy:
really appreciate!!!


#2

this is my logstash.conf .


(Magnus Bรคck) #3

Please don't post screenshots. Use copy/paste.


#4

:sweat: ok....


(Guy Boertje) #5

Do we presume you are using the Filebeat config from here


(Guy Boertje) #6

@houfan

You only get one event because the the second one's lines are being buffered internally - filebeat can only know when a previous set of lines is fully collected when the first line of the next set is seen.


#7

after so many times I have tried,I figured out the reason is ๏ผš

if [message] =~ "^# administrator" {
drop {}
}
if [message] =~ "^# Time" {
drop {}
}

this part didn't work right .It will drop much more messages than expect. cos when I delete this line

Time: 161214 11:32:21

and then logstash will give the correct two results.
I add this grok pattern at the end :(?:# Time:.*\n).

> (?m)^# User@Host: %{USER:User}\[[^\]]+\] @ (?:(?<clienthost>\S*) )?\[(?:%{IP:Client_IP})?\]\s+Id:\s+%{NUMBER:row_id:int}\n# Schema: %{WORD:Schema}\s+Last_errno: %{NUMBER}\s+Killed: %{NUMBER}\n# Query_time: %{NUMBER:Query_Time:float}\s+Lock_time: %{NUMBER:Lock_Time:float}\s+Rows_sent: %{NUMBER:Rows_Sent:int}\s+Rows_examined: %{NUMBER:Rows_Examined:int}\s+Rows_affected: %{NUMBER:Rows_affected:int}\n(?:# Bytes_sent: %{NUMBER:Byte_sent:int}\n)+(?:use %{DATA:database};\s*\n)?SET\s+timestamp=%{NUMBER:timestamp};(?:\n)?(?<sql>(?<action>\w+)\b.*;)?(?:\n)?(?:# Time:.*\n)?

logstash drop the first line โ€œ# Time: 161214 11:32:21โ€,and the whole messages (that the grok pattern matched )next.
at the end logstash only give one result.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.