Custom patterns using patterns_dir is not working

(Ranganath Nangineni) #1

I have the below custom patterns under /usr/share/logstash/patterns and /etc/logstash/patterns directories. These were created to accept HTTPDUSER in the COMBINEDPATCHELOG pattern.

The pattern is as below

USERNAME [a-zA-Z0-9.-]+
COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

Grok filter is as below:

if [fields][app] == "simple" {

    grok {
       patterns_dir => ["/usr/share/logstash/patterns" , "/etc/logstash/patterns"]
       match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}
       #match => { "message" => "%{SIMPLEAPACHELOG}" }
       #match => { "message" => "%{COMBINEDAPACHELOG}" }

But it is not accepting the format. I doubts it is not even looking into the patterns directory / overlooked by the existing default patterns.

Patterns core plugin:

logstash-patterns-core (4.1.2)
logstash-filter-grok (3.4.3)

The same pattern worked fine with the

Log Pattern: - [23/Jan/2018:22:35:51 -0800] "GET /content/resources/layouts/Trays/Sample/menu_bar_divider.gif HTTP/1.1" 200 1234 "" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" "" "image/gif"

(Magnus B├Ąck) #2
  match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}

Multiple problems:

  • You can't have double quotes inside a double quoted string. Make the string single-quoted.

  • Remove the trailing space.

  • The QS pattern includes the surrounding double quotes so unless you want to match ""foo"" you should say %{QS} instead of "%{QS}".

    match => { "message" => '%{COMBINEDAPACHELOG} "%{IP:trueclientip}" %{QS:filetype}'}

(Ranganath Nangineni) #3

I have placed the below pattern in the pattern file and then tried this pattern in Grok,

SIMPLEAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} "%{IP:trueclientip}" "%{GREEDYDATA:filetype}"

But it is not parsing the trueclientip and the filetype.

grok {
patterns_dir => ["/usr/share/logstash/patterns" , "/etc/logstash/patterns"]
#match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}
match => { "message" => "%{SIMPLEAPACHELOG}" }
#match => { "message" => "%{COMBINEDAPACHELOG}" }

so the ' (single quote ) will work as per your input ?

match => { "message" => '%{COMBINEDAPACHELOG} "%{IP:trueclientip}" %{QS:filetype}'}

(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.