Logstash Parsing Help

Elastic Stack Logstash
1

I am trying to parse NASA IIS logs. I get the error below:

[2017-11-16T16:48:48,905][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-11-16T16:48:48,912][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-16T16:48:49,189][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Expected one of #, => at line 55, column 17 (byte 1278) after filter {\n\tif [type] == \"iis\"{\n\n\t\tgrok {\n\t\t\tmatch => { \"message\" => \"^%{DATA:Host}\\s%{DATA:FIELD1}\\s%{DATA:FIELD2}\\s\\[%{DATA:TimeStamp}*\\]\\s\\\"%{WORD:Method}\\s%{DATA:Query}\\s%{DATA:HTTPVersion}\\\"\\s%{BASE10NUM:HTTPReply}\\s%{NUMBER:Bytes}?$\"\n\t\t}\n\t}\n}\n\noutput {\n  elasticsearch "}

Here is my grok filter:

filter {
	if [type] == "iis"{

		grok {
			match => { "message" => "^%{DATA:Host}\s%{DATA:FIELD1}\s%{DATA:FIELD2}\s\[%{DATA:TimeStamp}*\]\s\"%{WORD:Method}\s%{DATA:Query}\s%{DATA:HTTPVersion}\"\s%{BASE10NUM:HTTPReply}\s%{NUMBER:Bytes}?$"
		}
	}
}

My issue just seems to be incorrect syntax, however I am quite new to logstash so can't see where I am going wrong.

Thanks in advance

G

2

I don't think the escapes of the double quotes are handled well. Remove the backslashes and make the whole grok expression single-quoted instead.

Also, you really really don't want to overuse DATA patterns like that. You can get incorrect matches and really poor performance. In most cases you'll be able to use the far more efficient NOTSPACE instead.

3

Hi, I completely forgot to include an example of a message I am trying to parse. Please see below:

burger.letters.com - - [01/Jul/1995:00:00:12 -0400] "GET /images/NASA-logosmall.gif HTTP/1.0" 304 0

So I would need to escape all of the quotes referenced inside the grok pattern?

Is this how you would implement the NOTSPACE grok:
^%{NOTSPACE:Host}\s%{NOTSPACE:FIELD1}\s%{NOTSPACE:FIELD2}\s\[%{NOTSPACE:TimeStamp}*\]\s\"%{WORD:Method}\s%{NOTSPACE:Query}\s%{NOTSPACE:HTTPVersion}\"\s%{BASE10NUM:HTTPReply}\s%{NUMBER:Bytes}?$

Cheers,

G

4

So I would need to escape all of the quotes referenced inside the grok pattern?

No, don't escape them. Remove the escaping backslashes and make the string single-quoted.

Is this how you would implement the NOTSPACE grok:

Yes, that looks reasonable.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.