Custom patterns using patterns_dir is not working

Elastic Stack Logstash
1

I have the below custom patterns under /usr/share/logstash/patterns and /etc/logstash/patterns directories. These were created to accept HTTPDUSER in the COMBINEDPATCHELOG pattern.

The pattern is as below

USERNAME [a-zA-Z0-9.-]+
USER %{USERNAME}
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9.+-=:]+
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
HTTPDUSER %{EMAILADDRESS}|%{USER}
COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
SIMPLEAPACHELOG %{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{GREEDYDATA:filetype}"

Grok filter is as below:

if [fields][app] == "simple" {

    grok {
       patterns_dir => ["/usr/share/logstash/patterns" , "/etc/logstash/patterns"]
       match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}
       #match => { "message" => "%{SIMPLEAPACHELOG}" }
       #match => { "message" => "%{COMBINEDAPACHELOG}" }
    }

But it is not accepting the format. I doubts it is not even looking into the patterns directory / overlooked by the existing default patterns.

Patterns core plugin:

logstash-patterns-core (4.1.2)
logstash-filter-grok (3.4.3)

The same pattern worked fine with the grokconstructor.appspot.com/ https://grokdebug.herokuapp.com/

Log Pattern:

151.71.120.51 - Jacob@gmail.com [23/Jan/2018:22:35:51 -0800] "GET /content/resources/layouts/Trays/Sample/menu_bar_divider.gif HTTP/1.1" 200 1234 "https://Sample.xyz.com/site/mktg/gic/ITK/index.html" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" "10.17.158.213" "image/gif"

2 
  match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}

Multiple problems:

  • You can't have double quotes inside a double quoted string. Make the string single-quoted.

  • Remove the trailing space.

  • The QS pattern includes the surrounding double quotes so unless you want to match ""foo"" you should say %{QS} instead of "%{QS}".

    match => { "message" => '%{COMBINEDAPACHELOG} "%{IP:trueclientip}" %{QS:filetype}'}

3

I have placed the below pattern in the pattern file and then tried this pattern in Grok,

SIMPLEAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} "%{IP:trueclientip}" "%{GREEDYDATA:filetype}"

But it is not parsing the trueclientip and the filetype.

grok {
patterns_dir => ["/usr/share/logstash/patterns" , "/etc/logstash/patterns"]
#match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}
match => { "message" => "%{SIMPLEAPACHELOG}" }
#match => { "message" => "%{COMBINEDAPACHELOG}" }
}

so the ' (single quote ) will work as per your input ?

match => { "message" => '%{COMBINEDAPACHELOG} "%{IP:trueclientip}" %{QS:filetype}'}

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.