I want to calculate the nginx http latency from event.original field/keyword

Elastic Stack Elastic Agent
12

Oh @Subrahmanyam_Veerank I just realized you are using Elastic Agent NOT filebeat nginx module... so the process is a bit different!

Also what version of the nginx integration, hopefully the latest 1.17.0

With Agent you can provide a Custom Pipeline to run at the end... I will show you this.... let me figure out a quick / clean way to do this...

We will just add our custom pipeline with the name, this is at the bottom of the OOTB nginx integration ingest pipeline

GET _ingest/pipeline/logs-nginx.access-1.17.0

At the bottom you can see...

  {
    "pipeline": {
      "name": "logs-nginx.access@custom",
      "ignore_missing_pipeline": true
    }
  }

So Just go to Kibana - Dev Tools and PUT this... then this will be called at the end of the OOTB pipeline It should work

This is just a simple GROK (well that is not so simple) which parses the event.orginal but only set the last field (sine the others are already set)

PUT _ingest/pipeline/logs-nginx.access@custom
{
  "description": "Custom Pipeline for parsing Nginx access logs",
  "processors": [
    {
      "grok": {
        "pattern_definitions": {
          "NGINX_HOST": "(?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR})(:%{NUMBER})?",
          "NGINX_NOTSEPARATOR": """[^	 ,:]+""",
          "NGINX_ADDRESS_LIST": """(?:%{IP}|%{WORD})("?,?\s*(?:%{IP}|%{WORD}))*"""
        },
        "ignore_missing": true,
        "field": "event.original",
        "patterns": [
         "(%{NGINX_HOST} )?\"?(?:%{NGINX_ADDRESS_LIST}|%{NOTSPACE}) - (-|%{DATA}) \\[%{HTTPDATE}\\] \"%{DATA}\" %{NUMBER} %{NUMBER} \"(-|%{DATA})\" \"(-|%{DATA})\" (-|%{NUMBER:nginx.access.request_time:float})"
        ]
      }
    }
  ]
}

You can test with

POST _ingest/pipeline/logs-nginx.access@custom/_simulate
{
  "docs": [
    {
      "_source": {
        "@timestamp": "2023-12-29T18:19:51.218Z",
        "event": {
          "original": """172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486"""
        }
      }
    }
  ]
}

# Result

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "@timestamp": "2023-12-29T18:19:51.218Z",
          "event": {
            "original": """172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486"""
          },
          "nginx": {
            "access": {
              "request_time": 0.486
            }
          }
        },
        "_ingest": {
          "timestamp": "2024-01-03T21:11:12.105621943Z"
        }
      }
    }
  ]
}
1 Like