How to extract response and key-values value from response time string of apache logs

Elastic Stack Logstash
1

My logs are as follows :

::ffff:10.67.0.179 - - [23/Feb/2021:13:55:18 +0000] "GET /files/77570035-bc7e-4be7-9554-e2164dd9397e.otf HTTP/1.1" 200 211 "-" "-" "c37004e0-75de-11eb-b6d4-cb790f9fe1ad" "40.324 ms" "serviceName=file-download-service"

I have created a pipeline that works but the issue is that I am unable to get rid of "ms " and convert the response time to float , from response time field that is "40.324 ms". Also I am not able to separate "serviceName=file-download-service" field as key value name as I am getting complete value.

Here is the logstash pipeline :

input {
  file {
    path => "/Users/learnelk/Documents/logging/logstash/event-data/access.log"
  }
}

filter {


  grok {
    match => { "message" => "%{COMBINEDAPACHELOG} %{QS:coid} %{QS:responsetime} %{GREEDYDATA:sn}" }
  }
  mutate {
    remove_field => [ "message", "referrer", "agent" ]
    gsub => [
      "coid", '"', "",
      "responsetime", "ms", "",
      "responsetime", '"', "",
      "responsetime", ' ', "",
      "sn", '"', ""
    ]
    convert => {
      "response" => "integer"
      "bytes" => "integer"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Here is the output that I get :

{
              "sn" => "serviceName=file-download-service",
            "path" => "/Users/learnelk/Documents/logging/logstash/event-data/upload-access.log",
            "verb" => "GET",
           "bytes" => 211,
     "httpversion" => "1.1",
        "@version" => "1",
        "clientip" => "::ffff:10.67.0.179",
        "response" => 200,
    "responsetime" => "40.324",
            "coid" => "c37004e0-75de-11eb-b6d4-cb790f9fe1ad",
         "request" => "/files/77570035-bc7e-4be7-9554-e2164dd9397e.otf",
      "@timestamp" => 2021-06-04T12:15:08.604Z,
            "host" => "learnelk-mac.local",
            "auth" => "-",
           "ident" => "-",
       "timestamp" => "23/Feb/2021:13:55:18 +0000"
}

Kindly help .

2

You can use mutate+convert to convert [responsetime] to a float, and you can use a kv filter to parse [sn].

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.