Problem grok pattern

Elastic Stack Logstash
1

Hi all,

I'm issuing a problem while using filter grok. For instance, i'm trying to parse nginx logs using this pattern:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response}  (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{Q    S:referrer}) %{QS:agent} %{QS:xforwardedfor} %{IPORHOST:host} %{BASE10NUM:request_duration}

I'm pretty sure other people use this pattern for nginx log, it's pretty standard.

Because grok doesn't ship with this pattern by default, i'm using this configuration:

   filter{
           if [type] == "proxy_nginx" {
                   grok {
                           patterns_dir => "/etc/logstash/patterns"
                           match => { "message" => "%{NGINXACCESS}"}
                           add_field => {"index_name" => "nginx"}
                           add_tag => ["output_elastic"]
             }
           }
  }

I'm using the patterns_dir to tell logstash where to look for the pattern... it's not working and I have _grokparsefailure...

Here is an example log:

10.55.6.104 - - [2017-04-12T16:31:07+00:00] \"GET /check HTTP/1.1\" 200 0 \"-\" \"ELB-HealthChecker/2.0\"

2

I would test your custom pattern with https://grokdebug.herokuapp.com

see if it works there, though it is odd to see " in the log lines normally it is just a " and I presume that will not match

3

sorry, this is the log, that one was the one already parsed by elasticsearch.

10.55.10.68 - - [2017-04-13T07:42:09+00:00] "GET /check HTTP/1.1" 200 0 "-" "ELB-HealthChecker/2.0"

4

I made it work, the error was in the pattern. Thanks @eperry for the suggestion.

but now my doubt is why in elasticsearch i have the field like this?

"agent": "\"ELB-HealthChecker/2.0\""

Why elastic is putting a \" before and after the agent name? thanks

5

nobody know why i have this output in elasticsearch?

6

I found the problem, is elasticsearch that automatically maps strings. So it is using \ for escaping the " and consider the agent a string.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.