Logstash grok filter not working

Elastic Stack Logstash
#1

Hi and thanks for reading.
I'm trying to do my own postfix/imapd logs filter.
This is what i made:
%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\['?(%{BASE10NUM:id})?\]?:? %{GREEDYDATA:message}

And this is my grok filter

filter {
    	grok {
    		match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\[?(%{BASE10NUM:id})?\]?:? %{GREEDYDATA:message}" }
 }

At grokdebug.herokuapp.com it works. But Logstash give me this:

{:timestamp=>"2016-09-19T13:18:07.660000-0300", :message=>"fetched an invalid config", :config=>"input {\r\n\tfile {\r\n\t\tpath => \"/var/log/maillog\"\r\n\t\ttype => \"maillog\"\r\n\t\tstart_position => \"beginning\"\r\n\t\tignore_older => 0\r\n\t}\r\n}\r\n\r\nfilter {\r\n\tgrok {\r\n\t\tmatch => { \"message\" => \"%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\\[?(%{BASE10NUM:id})?\\]?:? %{GREEDYDATA:message}\" }\r\n}\r\n\r\noutput {\r\n\telasticsearch {\r\n\t\thosts => [ \"localhost:9200\" ]\r\n\t}\r\n}\r\n\n", :reason=>"Expected one of #, => at line 16, column 16 (byte 342) after filter {\r\n\tgrok {\r\n\t\tmatch => { \"message\" => \"%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\\[?(%{BASE10NUM:id})?\\]?:? %{GREEDYDATA:message}\" }\r\n}\r\n\r\noutput {\r\n\telasticsearch ", :level=>:error}

Hope this helps to understand. Thanks!

#2

It looks like there's a closing brace missing; you're never closing the filter block.

1 Like
#3

Thanks for reply. I fixed this problem just doing what you said.
I keep going on with this. Now i have another issue (i don't want to open a new topic).
I have this filter:
> filter {
> grok {
> match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?[?(%{BASE10NUM:id})?]?:?} %{WORD:action}%{GREEDYDATA:action_info}" }
> }
> if [action] == "disconnect" or [action] == "connect" {
> grok {
> match => { "action_info" => " from %{GREEDYDATA:host}[%{IPV4:ip_host}]" }
> }
> }
> }

And it is not working. Logstash add the tag "_grokparsefailure" for each log.
..."tags":["_grokparsefailure"]}

#4

Please show the whole event as produced by a stdout { codec => rubydebug } output.

#5

The stdout { codec => rubydebug } output is too long to put here, but is extremely repetitive, so i copy a paragraph:

{
       "message" => "Sep 18 04:40:15 mail postfix/smtpd[1979]: connect from localhost[127.0.0.1]",
      "@version" => "1",
    "@timestamp" => "2016-09-19T20:15:53.638Z",
          "path" => "/var/log/dsaasd",
          "host" => "ELK-System",
          "type" => "registro_mail",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}
#6

Multiple problems, mainly here: [?(%{BASE10NUM:id})?]?:?}

  • The square brackets must be escaped.
  • Why all the question marks everywhere?
  • What's up with the closing brace at the end?

Suggestion if the bracketed pid should be optional: (\[%{BASE10NUM:id}\])?

When debugging things like this, always start with a short expression, in this case e.g. %{SYSLOGTIMESTAMP:timestamp}, then add more and more until things break.

#7