Need help ignoring certain lines in Grok filter


#1

I have a custom Grok filter that matches our mysql slow logs, but our slow logs have extra output at the beginning and end of the file. I need to have Grok ignore these entries. This is an example of our slow log along with my Grok filter. The filter works fine, I just need to know how to ignore the extra data:

Grok Filter:
^# User@Host: %{USER:[mysql][slowlog][user]}(\[[^\]]+\])? @ %{HOSTNAME:[mysql][slowlog][host]} %{SYSLOG5424SD:[mysql][slowlog][ip]}\n^# (Thread_id: %{NUMBER:[mysql][slowlog][id]})?.*Schema: %{WORD:[mysql][slowlog][schema]}.*QC_hit: %{WORD:[mysql][slowlog][is_qc_hit]}\n^# Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]}\n(SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]};\n)?%{GREEDYMULTILINE:[mysql][slowlog][query]}

Custom GREEDYMULTILINE pattern:
(.|\n)*

Example slow log query:

# User@Host: user[user] @ host.com [127.0.0.1]
# Thread_id: 437308935 Schema: db QC_hit: No
# Query_time: 3.375738 Lock_time: 0.000030 Rows_sent: 1 Rows_examined: 1
SET timestamp=1512977437;
SELECT * FROM example_table WHERE uid = 1234;

This is the excess data that I need to ignore. It only shows up at the beginning and end of our logs so it could appear before or after a query.If there's a Grok pattern that could ignore any occurrence of this extra data that would be best

/usr/libexec/mysqld, Version: 5.5.56-MariaDB (MariaDB Server). started with:
Tcp port: 3306 Unix socket: /data/mysql/mysql.sock
Time Id Command Argument

Any help would be greatly appreciated


(Sjaak) #2

The excess data doesn't match your grok pattern right?

In that case you could add something like this to the end of your grok

tag_on_failure => [ "_grok_failure" ]

and then

if "_grok_failure" in [tags] {
drop { }
}


#3

The problem is that this pattern is being included in my GREEDYMULTILINE matching. So it grabs the beginning of the query and everything else between until it matches the start of the grok pattern. I need to exclude this pattern from GREEDYMULTILINE


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.