”_grokparsefailure” even though the grok pattern works

Elastic Stack Logstash
1

I am trying to parse different logs lines from two different type of file : slave and master. I did test my pattern in the Grok Dubugger and it is working fine but tags field in kibana is _grokparsefailure.

Here is my config file

input {
	file { 
		type => "slave"
		path => "/home/mathis/Documents/**/intranet*.log"
		exclude =>"*8402.log"
		sincedb_path => '/dev/null'
		start_position => beginning
	}
	file { 
		type => "master"
		path => "/home/mathis/Documents/**/intranet*8402.log"
		sincedb_path => '/dev/null'
	}
}
filter {
	if [type] == "slave" {
		grok {
			match => { "message" => ["\[%{DATESTAMP:eventtime}\] \- %{USERNAME:user} \- %{IPV4:clientip} \- %{NUMBER} \- %{WORD} %{NUMBER:exectime} %{WORD} %{NUMBER:time} %{GREEDYDATA:data} %{NUMBER:waittime}"] }
			remove_field => "message"
		}
		grok {
			match => { "message" => ["\[%{DATESTAMP:eventtime}\] \- Process status database sync \- %{WORD}\.%{WORD}\.%{WORD}\:%{NUMBER:slavenumb}\(\#%{NUMBER}\) \(load %{NUMBER:nbutilisateur} grace period 5 minutes\) %{GREEDYDATA}"] }
			remove_field => "message"
		}
	date {
        		match => [ "eventtime", "dd/MM/YYYY HH:mm:ss.SSS" ]
			target => "@timestamp"
 		}
	}
	if [type] == "master" {
		grok {
        		match => {"message" => ["%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}(?<starttime>((?!<[0-9])%{HOUR}:)?%{MINUTE}(?::%{SECOND})(?![0-9]))"]}
        		remove_field => "message"
		}
    		date {
        		match => [ "starttime", "HH:mm:ss","mm:ss" ]
    		}
	}
		
if "_grokparsefailure" in [tags] {
  		drop { }
	}	
}
output {
	elasticsearch {
		hosts => "127.0.0.1:9200"
		index => "logstash-local3-%{+YYYY.MM.dd}"
	}
}

Here are the 3 logs lines that I want to parse :
(they are in the order of groks in my conf file)

[24/06/2020 21:57:29.548] - Process status database sync - us1salx08167.corpnet2.com:8100(#53738) (load 0 grace period 5 minutes) : current date 2020/06/24 21:57:29 update date 2020/06/24 21:55:44 old state OK new state OK

[29/05/2020 07:41:51.354] - ih912865 - 10.104.149.128 - 93 - Transaction 7635 COMPLETED 318 ms wait time 3183 ms

   31730  31626  464 10970020     52:25 /plw/modules/bin/Lx86_64/opx2-intranet.exe -I /plw/modules/bin/Lx86_64/opx2-intranet.dxl -H /plw/modules/bin/Lx86_64 -L /plw/PLW_PROD/modules/preload-intranet.ini -- plw-sysconsole -port 8400 -logdir /plw/PLW_PROD/httpdocs/admin/log/ -slaves 2
2

For the slave logs you have two groks, even if one matches the other will produce a _grokparsefailure.

grok can match against an array of patterns, so you could combine the two into one.

1 Like
3

Ok thanks, that's a point, can you show me the syntax ?
But there is an other problem because I have _grokparsefailure for every line.

4 
if [type] == "slave" {
	grok {
		match => { "message" => [
                        "\[%{DATESTAMP:eventtime}\] \- %{USERNAME:user} \- %{IPV4:clientip} \- %{NUMBER} \- %{WORD} %{NUMBER:exectime} %{WORD} %{NUMBER:time} %{GREEDYDATA:data} %{NUMBER:waittime}",
                        "\[%{DATESTAMP:eventtime}\] \- Process status database sync \- %{WORD}\.%{WORD}\.%{WORD}\:%{NUMBER:slavenumb}\(\#%{NUMBER}\) \(load %{NUMBER:nbutilisateur} grace period 5 minutes\) %{GREEDYDATA}"
                    ] }
		remove_field => "message"
	}
	...
1 Like
5

Thanks for the upgrade. Unfortunately this does not solve the problem of _grokparsefailure.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.