Grok filter not adding new fields

Elastic Stack Logstash
1

We have the below grok filter configured for our journlabeat. The same was deployed on our local for filebeat was working fine but isn't adding the new fields on journalbeat.

filter {

    grok {
      patterns_dir => ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns"]
      match => { "message" => [
        '%{IPV4:client_ip} - - \[%{HTTPDATE:date}\] "%{WORD:method} %{URIPATH:request} %{URIPROTO:protocol}\/[1-9].[0-9]" (%{NUMBER:status}|-) (%{NUMBER:bytes}|-) "(%{URI:url}|-)" %{QUOTEDSTRING:client}'
        ]
        break_on_match => false
        tag_on_failure => ["failed_match"]
      }
    }
}

We tried adding the mutate filter for adding new fields using below but it isn't fetching the value and is printing the scalar values itself (example: %{client_ip}).

mutate {
        add_field => {
           "client_ip" => "%{client_ip}"
           "date" => "%{date}"
           "method" => "%{method}"
           "status" => "%{status}"
           "request" => "%{request}"
        }
      }

The log which we are trying to match is as below.

::ffff:172.65.205.3 - - [09/Jul/2020:11:32:52 +0000] "POST /v1-get--profile HTTP/1.1" 404 71 "https://mycompany.com/customer/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"

Could someone let me know what exactly are we doing wrong. Thanks in Advance.

2

Hi,

What not using this grok pattern?
HTTPD_COMBINEDLOG
found here:

3

The closing curly bracket for your match option should be moved two lines further to the top because right now it also wraps the next two options of your grok filter.

(And you can definitely delete the mutate filter. Right now it basically says: Create a field called X and fill it with the content of the field X. That will never be a helpful operation :slight_smile: )

Edit: Ooh. I had not realized that Rom1 had digged out such an old thread :upside_down_face:

4

Thank you guys for the response. This was a while ago and is fixed. A bit embarrassing that I don't remember what was the scenario back then. But we updated our grok to below and everything is working fine now. Posting it here, if anyone else lands here searching for the same. :slight_smile:

grok {
      patterns_dir => ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns"]
      match => { "message" => [
        '%{IPV4:client_ip} - - \[%{HTTPDATE:date}\] "%{WORD:method} %{URIPATH:request}(%{URIPARAM:uriparam}|) %{URIPROTO:protocol}\/[1-9].[0-9]" (%{NUMBER:status}|-) (%{NUMBER:bytes}|-) "(%{URI:url}|-)" %{QUOTEDSTRING:client}'
        ]
        break_on_match => false
        tag_on_failure => ["failed_match"]
      }
    }
    mutate {
      convert => {
        method => "string"
        status => "integer"
        url => "string"
      }
    }
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.