Gorkparsefailure

Elastic Stack Logstash
1

Dear all,

I am getting gorkparsefailure error for my log lines. Kindly help, 70% of my datas are not getting structured.

Conf file
file{
path => ["/my/path/files/13_70/", "/my/path/files/13_71/"]
exclude => "*.gz"
start_position => "beginning"
}
}
filter {

	grok {
		match => 
		{
				"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE}\] "%{WORD} %{DATA} HTTP/%{NUMBER}" %{NUMBER} %{NUMBER} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE}'
		}
		remove_field => [ "port", "tags" ]
	}

	if [cliip] == "[" {
	
		grok {
			match => 
				{
					"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
				}
			remove_field => [ "port", "tags" ]
			}  
	}

	else {
		grok {
			match => 
				{
					"message" => '%{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
				}
				remove_field => [ "port", "tags" ]
			}
		}
}

output {

if "_grokparsefailure" in [tags] {
# write events that didn't match to a file
	file { 
	"path" => "/mypath/exports/grok_failures.txt" 
	}
} 

else {
	elasticsearch{
		hosts => "1.1.1.1"
		index => "index1"
	}
}

}

2

No one can help you if you don't proivde an example of a log line which produces _grokparsefailure :slight_smile:

EDIT: Thanks!

3

my gorlfailuretxt file

{"@timestamp":"2019-01-03T05:45:25.039Z","@version":"1","tags":["_grokparsefailure"],"message":"10.10.10.10:56384 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 6108 - myserver-ABCD-CD7:9080 -","path":"/my/path/files/13_70/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.040Z","@version":"1","tags":["_grokparsefailure"],"message":"11.11.11.11:28403 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 2701 - myserver-ACD2-CL4:9080 -","path":"/my/path/files/13_71/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.048Z","@version":"1","tags":["_grokparsefailure"],"message":"12.12.12.12:34305 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 6252 - myserver-ACD2-CD7:9080 -","path":"/my/path/files/13_71/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.048Z","@version":"1","tags":["_grokparsefailure"],"message":"10.10.01.10:40412 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 8776 - myserver-ACD2-CD7:9080 -","path":"/my/path/files/13_71/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.049Z","@version":"1","tags":["_grokparsefailure"],"message":"1.10.10.10:39458 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 2638 - myserver-ABCD-CD7:9080 -","path":"/my/path/files/13_71/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.049Z","@version":"1","tags":["_grokparsefailure"],"message":"42.11.11.1:38165 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 7861 - myserver-ACD2-CD7:9080 -","path":"/my/path/files/13_71/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.128Z","@version":"1","tags":["_grokparsefailure"],"message":"[1009:3770:3510:f5f3:6920:b932:68f5:88c0], 163.154.112.52:25320 - - - [26/Dec/2018:02:31:02 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 127432 - myserver-ACD2-CL4:9080 -","path":"/my/path/files/13_71/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.142Z","@version":"1","tags":["_grokparsefailure"],"message":"[1073:204:a040:f39::2b37:98b1], 163.154.112.52:39990 - - - [26/Dec/2018:02:31:02 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 3852 - myserver-ACD2-CL4:9080 -","path":"/my/path/files/13_70/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.143Z","@version":"1","tags":["_grokparsefailure"],"message":"16.45.17.23:47320 - - - [26/Dec/2018:02:31:02 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 3588 - myserver-ACD2-CL4:9080 -","path":"/my/path/files/13_70/access_log.2018-12-26-02_15_00","host":"myhst"}

{"@timestamp":"2019-01-03T05:45:25.144Z","@version":"1","tags":["_grokparsefailure"],"message":"1.78.35.75:57395 - - - [26/Dec/2018:02:31:02 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 4534 - myserver-ACD2-CL4:9080 -","path":"/my/path/files/13_70/access_log.2018-12-26-02_15_00","host":"myhst"}

4

kindly help me why am i getting gork parse failure?
What i have change in my conf file

5

my two different log lines are

1. 42.11.11.1:38165 - - - [26/Dec/2018:02:31:01 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 7861 - myserver-ACD2-CD7:9080 -

2.[1009:3770:3510:f5f3:6920:b932:68f5:88c0], 163.154.112.52:25320 - - - [26/Dec/2018:02:31:02 +0530] "POST /wls62/apps/services/configprofile HTTP/1.1" 204 - - 127432 - myserver-ACD2-CL4:9080 -

6

@magnusbaeck Could you please help me in this issue.
How to read two different line of logs. I tried many ways but still 70% logs are not getting structured.

Another thread : How to check the tags values in logstash

Kindly guide me how to proceed further.

Thanks in Advance

7

@admlko any luck?

8

I do not have time to go through your grok patterns, but would recommend starting to build them out from the start as explained in this blog post. Be aware that you can also list multiple patterns for a match and Logstash will go through them until it finds one that matches, which can simplify your configuration and remove the. need for conditionals around the grok patterns.

9

@Christian_Dahlqvist Thanks much.

But i have two URI path in my log line.
[Thu Nov 22 00:52:01 2018] [error] [client 16.51.30.205:172] File does not exist: /myapp/test/HTTPServer/htdocs/favicon.ico, referer: https://myapp.mycomp.com/response/response.do

In this case how to mange?

could you please also send me the link for gork regex tutorial for beginners

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.