COMBINEDAPACHELOG grok definition not creating the expected fields

Elastic Stack Logstash
1

Complete beginner to logstash here...

I'm following the "Parsing Logs with Logstash" tutorial. Input comes from filebeat reading a static log file provided with the tutorial; see example of a sample line below.

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

Following the tutorial, I'm using the COMBINEDAPACHELOG grok definition to parse and create some fields: "clientip, ident, auth, etc" that I can feed to other filters later on, but I don't get any of these fields created in the response.

Logstash is version 8.1.1 and so is filebeat.

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    stdout { codec => rubydebug }
}

Below is a sampled response I get:

{
           "log" => {
        "offset" => 1624,
          "file" => {
            "path" => "D:\\Elastic\\Logstash\\logstash-tutorial.log"
        }
    },
          "http" => {
         "request" => {
              "method" => "GET",
            "referrer" => "http://semicomplete.com/presentations/logstash-monitorama-2013/"
        },
         "version" => "1.1",
        "response" => {
                   "body" => {
                "bytes" => 430406
            },
            "status_code" => 200
        }
    },
    "user_agent" => {
        "original" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
    },
    "@timestamp" => 2022-03-24T13:18:23.996Z,
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1\" 200 430406 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
      "@version" => "1",
           "ecs" => {
        "version" => "8.0.0"
    },
          "host" => {
        "name" => "myhostname"
    },
         "input" => {
        "type" => "log"
    },
     "timestamp" => "04/Jan/2015:05:13:42 +0000",
        "source" => {
        "address" => "83.149.9.216"
    },
           "url" => {
        "original" => "/presentations/logstash-monitorama-2013/images/sad-medic.png"
    },
         "event" => {
        "original" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1\" 200 430406 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\""
    },
         "agent" => {
                "type" => "filebeat",
                  "id" => "88432da2-fbc0-4b35-9c31-8497c85b3ea4",
             "version" => "8.1.1",
                "name" => "myhostname",
        "ephemeral_id" => "23bee668-d950-4bb9-83c4-ca2714b468cd"
    }
}
2

When ecs_compatibility is enabled the grok patterns that are loaded will create [source][address] rather than [clientip] and so on. If compatibility is enabled at the global or pipeline level there is still an option to disable it at the filter level and get the old names.

3

Disabling the ecs_compatibility at the filter level yield the expected fields in the processed events. Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.