im trying to calculate the nginx http latency from event.original field.
event.original
172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486
For example, from the above log the latency value is 0.486 and i want to create a new field "latency" and the value should be extracted from the event.original. I tried to create scripted field and runtime field but im not able to get that. Kindly give the proper solution for this.
Did you try the filebeat nginx module? I think it will do that for you assuming your nginx logs use the default format.
The Filebeat Quick start shows how to set this up
As @stephenb mentioned, you can try default modules if you are using the nginx log pattern. If you are using any custom pattern you can also try ingest pipelines where you can write grok to extract the values.
We also just had a long discussion on custom nginx logs here
im using the nginx integration with elastic agent deployed on that server. in the logs-* index im getting this event.oringinal. But there is option in the nginx integration as processors so i need what i have to put there to get the value 0.486 from the below keyword event.original
172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486
Can you provide a sample of the JSON document from elasticsearch that is parsed / show all the fields... I thought that field was automatically parsed by the nginx integration.
Please find the json output below
{
"@timestamp": [
"2024-01-03T05:51:46.000Z"
],
"agent.ephemeral_id": [
"a37c65e9-3287-452c-9e64-1e39146da7d2"
],
"agent.id": [
"ee54e645-cb2a-48e2-8eb3-d2d9236bb5ef"
],
"agent.name": [
"FSIGW"
],
"agent.type": [
"filebeat"
],
"agent.version": [
"8.11.0"
],
"data_stream.dataset": [
"nginx.access"
],
"data_stream.namespace": [
"default"
],
"data_stream.type": [
"logs"
],
"ecs.version": [
"8.5.1"
],
"elastic_agent.id": [
"ee54e645-cb2a-48e2-8eb3-d2d9236bb5ef"
],
"elastic_agent.snapshot": [
false
],
"elastic_agent.version": [
"8.11.0"
],
"event.agent_id_status": [
"verified"
],
"event.category": [
"web"
],
"event.created": [
"2024-01-03T05:51:47.274Z"
],
"event.dataset": [
"nginx.access"
],
"event.ingested": [
"2024-01-03T05:51:47.000Z"
],
"event.kind": [
"event"
],
"event.module": [
"nginx"
],
"event.original": [
"182.28.12.42 - - [03/Jan/2024:11:21:46 +0530] \"POST /cbsprod/requestJson HTTP/1.1\" 200 1118 \"-\" \"Go-http-client/1.1\" 0.179"
],
"event.outcome": [
"success"
],
"event.timezone": [
"+05:30"
],
"event.type": [
"access"
],
"host.architecture": [
"x86_64"
],
"host.containerized": [
false
],
"host.hostname": [
"fsigw"
],
"host.id": [
"312ef035755047259e5184d2c4d0d73c"
],
"host.ip": [
"182.28.12.106",
"fe80::215:5def:fe4c:3e1f",
"182.17.0.1"
],
"host.mac": [
"00-13-5D-4C-3E-1F",
"02-42-1C-69-B1-97"
],
"host.name": [
"fsigw"
],
"host.os.codename": [
"focal"
],
"host.os.family": [
"debian"
],
"host.os.kernel": [
"5.4.0-169-generic"
],
"host.os.name": [
"Ubuntu"
],
"host.os.name.text": [
"Ubuntu"
],
"host.os.platform": [
"ubuntu"
],
"host.os.type": [
"linux"
],
"host.os.version": [
"20.04.6 LTS (Focal Fossa)"
],
"http.request.method": [
"POST"
],
"http.response.body.bytes": [
1118
],
"http.response.status_code": [
200
],
"http.version": [
"1.1"
],
"input.type": [
"log"
],
"log.file.path": [
"/var/log/nginx/access.log"
],
"log.offset": [
224505906
],
"nginx.access.remote_ip_list": [
"182.28.12.42"
],
"related.ip": [
"182.28.12.42"
],
"source.address": [
"182.28.12.42"
],
"source.ip": [
"182.28.12.42"
],
"tags": [
"preserve_original_event",
"nginx-access"
],
"url.original": [
"/cbsprod/requestJson"
],
"url.original.text": [
"/cbsprod/requestJson"
],
"url.path": [
"/cbsprod/requestJson"
],
"user_agent.device.name": [
"Other"
],
"user_agent.name": [
"Go-http-client"
],
"user_agent.original": [
"Go-http-client/1.1"
],
"user_agent.original.text": [
"Go-http-client/1.1"
],
"user_agent.version": [
"1.1"
],
"_id": "8K_gzYwBYj5GlUZ8vqjJ",
"_index": ".ds-logs-nginx.access-default-2023.12.22-000002",
"_score": null
}
Hmmmm interesting... I thought response time was one of the OOTB extracted fields but it looks like It is not or you have a custom format.
I will look tomorrow but we will probably need to do What I showed in the other post I referenced above... Update the ingest pipeline to extract that field that's the right way to do it.
Will check tomorrow then.
Do you have multiple formats for the log or are all logs this format....
172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486
If this is always your format then you just need to add this to the end of the grok statement and you can follow the instructions in the post I posted above.
Oh @Subrahmanyam_Veerank I just realized you are using Elastic Agent NOT filebeat nginx module... so the process is a bit different!
Also what version of the nginx integration, hopefully the latest 1.17.0
With Agent you can provide a Custom Pipeline to run at the end... I will show you this.... let me figure out a quick / clean way to do this...
We will just add our custom pipeline with the name, this is at the bottom of the OOTB nginx integration ingest pipeline
GET _ingest/pipeline/logs-nginx.access-1.17.0
At the bottom you can see...
{
"pipeline": {
"name": "logs-nginx.access@custom",
"ignore_missing_pipeline": true
}
}
So Just go to Kibana - Dev Tools and PUT this... then this will be called at the end of the OOTB pipeline It should work
This is just a simple GROK (well that is not so simple) which parses the event.orginal but only set the last field (sine the others are already set)
PUT _ingest/pipeline/logs-nginx.access@custom
{
"description": "Custom Pipeline for parsing Nginx access logs",
"processors": [
{
"grok": {
"pattern_definitions": {
"NGINX_HOST": "(?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR})(:%{NUMBER})?",
"NGINX_NOTSEPARATOR": """[^ ,:]+""",
"NGINX_ADDRESS_LIST": """(?:%{IP}|%{WORD})("?,?\s*(?:%{IP}|%{WORD}))*"""
},
"ignore_missing": true,
"field": "event.original",
"patterns": [
"(%{NGINX_HOST} )?\"?(?:%{NGINX_ADDRESS_LIST}|%{NOTSPACE}) - (-|%{DATA}) \\[%{HTTPDATE}\\] \"%{DATA}\" %{NUMBER} %{NUMBER} \"(-|%{DATA})\" \"(-|%{DATA})\" (-|%{NUMBER:nginx.access.request_time:float})"
]
}
}
]
}
You can test with
POST _ingest/pipeline/logs-nginx.access@custom/_simulate
{
"docs": [
{
"_source": {
"@timestamp": "2023-12-29T18:19:51.218Z",
"event": {
"original": """172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486"""
}
}
}
]
}
# Result
{
"docs": [
{
"doc": {
"_index": "_index",
"_version": "-3",
"_id": "_id",
"_source": {
"@timestamp": "2023-12-29T18:19:51.218Z",
"event": {
"original": """172.29.55.40 - - [27/Dec/2023:10:08:14 +0530] "POST /prod/requestJson HTTP/1.1" 200 607 "-" "Go-http-client/1.1" 0.486"""
},
"nginx": {
"access": {
"request_time": 0.486
}
}
},
"_ingest": {
"timestamp": "2024-01-03T21:11:12.105621943Z"
}
}
}
]
}
1 Like
Thanks sir. it is working. Can you please guide me how to write the GROK and extract the desired fields from the log entry incase of our future requirements? Please suggest any reference to learn the GROK
Grok processoredit
Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.
This processor comes packaged with many reusable patterns.
If you need help building patterns to match your logs, you will find the Grok Debugger tool quite useful! The Grok Constructor is also a useful tool.
The Grok Constructor incremental constructor is very helpful.
Lots of online content...
@stephenb Hi sir..There is some issue in elastic agent on kubernetes but there is no reply from elastic side. Not able to get orchestrator.cluster.name and some info in kubernetes dashboards are not getting populated
Can you please forward this to the concerned team?
Field
Value
_id
_oT59owBL1hahzUj_psT
_index
.ds-logs-nginx_ingress_controller.access-default-2024.01.10-000001
_score
@timestamp
Jan 10, 2024 @ 15:44:00.000
agent.ephemeral_id
6e25e07e-9522-4941-ab10-4079d018de6f
agent.id
4badf0ef-fd8d-4e8f-9b5c-ae416e3c94f0
agent.name
K8SRW7
agent.type
filebeat
agent.version
8.11.0
container.id
4edcde814ca666ba6d69f84e670cb0f438ad9d770cc172d5829df957f0ac453d
container.image.name
registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd
container.runtime
docker
data_stream.dataset
nginx_ingress_controller.access
data_stream.namespace
default
data_stream.type
logs
ecs.version
8.4.0
elastic_agent.id
4badf0ef-fd8d-4e8f-9b5c-ae416e3c94f0
elastic_agent.snapshot
false
elastic_agent.version
8.11.0
event.agent_id_status
verified
event.category
web
event.created
2024-01-10T10:14:00.796Z
event.dataset
nginx_ingress_controller.access
event.ingested
Jan 11, 2024 @ 10:53:47.000
@stephenb sir..when i checked the nginx ingress controller access logs there is huge difference in event.created and event.ingested. Can you please intimate what is the issue?
Please show the JSON like you did above with the the _source and fields
Differences are usually from ingestion delayed or catching up differences and or Not Understanding / Using Timezones correctly or a combination of both.
You will need to look closely to determine what is going on.
event.created
2024-01-10T10:14:00.796Z <--- UTC
event.ingested
Jan 11, 2024 @ 10:53:47.000 <--- Probably Local Time Zone
One of those appears to be UTC the other Local you need to look at them both in UTC to determine the offset / delay
When i checked the logs yesterday there is delay of ingestion of around 10hrs and in the morning it was catched upto time and now again there is delay of 30-40min. Whether issue is with resources or application?
{
"@timestamp": [
"2024-01-12T04:54:20.000Z"
],
"agent.ephemeral_id": [
"06f7117f-5a52-40a8-b14e-c6d8b6b5149e"
],
"agent.id": [
"4badf0ef-fd8d-4e8f-9b5c-ae416e3c94f0"
],
"agent.name": [
"K8SRW7"
],
"agent.type": [
"filebeat"
],
"agent.version": [
"8.11.0"
],
"container.id": [
"4edcde814ca666b70cb0f438ad9d770cc172d5829df957f0ac453d"
],
"container.image.name": [
"registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c48293e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd"
],
"container.runtime": [
"docker"
],
"data_stream.dataset": [
"nginx_ingress_controller.access"
],
"data_stream.namespace": [
"default"
],
"data_stream.type": [
"logs"
],
"ecs.version": [
"8.4.0"
],
"elastic_agent.id": [
"4baef-fd8d-4e8f-9b5c-ae416e3c94f0"
],
"elastic_agent.snapshot": [
false
],
"elastic_agent.version": [
"8.11.0"
],
"event.agent_id_status": [
"verified"
],
"event.category": [
"web"
],
"event.created": [
"2024-01-12T04:54:20.940Z"
],
"event.dataset": [
"nginx_ingress_controller.access"
],
"event.ingested": [
"2024-01-12T05:26:01.000Z"
],
"event.kind": [
"event"
],
"event.module": [
"nginx_ingress_controller"
],
"event.outcome": [
"failure"
],
"event.timezone": [
"+00:00"
],
"event.type": [
"info"
],
"host.architecture": [
"x86_64"
],
"host.containerized": [
false
],
"host.hostname": [
"k8srw7"
],
"host.id": [
"363cf00252ab40d3b8ca818c588693e6"
],
"host.name": [
"k8srw7"
],
"host.os.codename": [
"focal"
],
"host.os.family": [
"debian"
],
"host.os.kernel": [
"5.4.0-167-generic"
],
"host.os.name": [
"Ubuntu"
],
"host.os.name.text": [
"Ubuntu"
],
"host.os.platform": [
"ubuntu"
],
"host.os.type": [
"linux"
],
"host.os.version": [
"20.04.6 LTS (Focal Fossa)"
],
"http.request.bytes": [
2248
],
"http.request.id": [
"f471794cfd036010e06afcbddcb8110b"
],
"http.request.method": [
"POST"
],
"http.response.body.bytes": [
150
],
"http.response.status_code": [
502
],
"http.version": [
"1.1"
],
"input.type": [
"filestream"
],
"kubernetes.container.name": [
"controller"
],
"kubernetes.labels.app_kubernetes_io/component": [
"controller"
],
"kubernetes.labels.app_kubernetes_io/instance": [
"myingress"
],
"kubernetes.labels.app_kubernetes_io/managed-by": [
"Helm"
],
"kubernetes.labels.app_kubernetes_io/name": [
"ingress-nginx"
],
"kubernetes.labels.app_kubernetes_io/part-of": [
"ingress-nginx"
],
"kubernetes.labels.app_kubernetes_io/version": [
"1.8.1"
],
"kubernetes.labels.helm_sh/chart": [
"ingress-nginx-4.7.1"
],
"kubernetes.labels.pod-template-hash": [
"74448676fc"
],
"kubernetes.namespace": [
"ingress-nginx-latest"
],
"kubernetes.namespace_labels.field_cattle_io/projectId": [
"p-v7jkq"
],
"kubernetes.namespace_uid": [
"7700e1dc-cad0-4276-b0a0-3069a047154e"
],
"kubernetes.node.hostname": [
"k8srw7"
],
"kubernetes.node.labels.beta_kubernetes_io/arch": [
"amd64"
],
"kubernetes.node.labels.beta_kubernetes_io/os": [
"linux"
],
"kubernetes.node.labels.kubernetes_io/arch": [
"amd64"
],
"kubernetes.node.labels.kubernetes_io/hostname": [
"k8srw7"
],
"kubernetes.node.labels.kubernetes_io/os": [
"linux"
],
"kubernetes.node.labels.node-role_kubernetes_io/worker": [
"true"
],
"kubernetes.node.labels.workload": [
"nginx-ingress"
],
"kubernetes.node.name": [
"k8srw7"
],
"kubernetes.node.uid": [
"eae48dd-40b8-8214-4ca7900d2b79"
],
"kubernetes.pod.ip": [
"xxxxxx"
],
"kubernetes.pod.name": [
"myingress-ingress-nginx-controller-74448676fc-qxxfb"
],
"kubernetes.pod.uid": [
"6c208-2a5e-4b5a-bb80-799576630780"
],
"kubernetes.replicaset.name": [
"myingress-ingress-nginx-controller-74448676fc"
],
"log.file.device_id": [
"64768"
],
"log.file.inode": [
"4849754"
],
"log.file.path": [
"/var/log/containers/myingress-ingress-nginx-controller-74448676fc-qxxfb_ingress-nginx-latest_controller-4edca666ba6d69f84e670cb0f438ad9d770cc172d5829df957f0ac453d.log"
],
"log.offset": [
34782764845
],
"nginx_ingress_controller.access.http.request.id": [
"f471794cfd036010e06afcbddcb8110b"
],
"nginx_ingress_controller.access.http.request.length": [
2248
],
"nginx_ingress_controller.access.http.request.time": [
0.002
],
"nginx_ingress_controller.access.remote_ip_list": [
"xxxxxxx"
],
"nginx_ingress_controller.access.upstream_address_list": [
"xxxxxxxx"],
"nginx_ingress_controller.access.upstream.alternative_name": [
""
],
"nginx_ingress_controller.access.upstream.ip": [
"xxxxxxxxx"
],
"nginx_ingress_controller.access.upstream.name": [
"testportal-service-9036"
],
"nginx_ingress_controller.access.upstream.port": [
9036
],
"nginx_ingress_controller.access.upstream.response.length": [
0
],
"nginx_ingress_controller.access.upstream.response.length_list": [
"0",
"0",
"0"
],
"nginx_ingress_controller.access.upstream.response.status_code": [
502
],
"nginx_ingress_controller.access.upstream.response.status_code_list": [
"502",
"502",
"502"
],
"nginx_ingress_controller.access.upstream.response.time": [
0.003
],
"nginx_ingress_controller.access.upstream.response.time_list": [
"0.001",
"0.001",
"0.001"
],
"related.ip": [
"xxxxxxx"
],
"source.address": [
"xxxxxxxx"
],
"source.as.number": [
38266
],
"source.as.organization.name": [
"Vodafone Idea Ltd"
],
"source.as.organization.name.text": [
"Vodafone Idea Ltd"
],
"source.geo.city_name": [
"Delhi"
],
"source.geo.continent_name": [
"Asia"
],
"source.geo.country_iso_code": [
"IN"
],
"source.geo.country_name": [
"India"
],
"source.geo.location": [
{
"coordinates": [
,
],
"type": "Point"
}
],
"source.geo.region_iso_code": [
"IN-DL"
],
"source.geo.region_name": [
"National Capital Territory of Delhi"
],
"source.ip": [
"xxxxxxx"
],
"stream": [
"stdout"
],
"tags": [
"nginx-ingress-controller-access"
],
"url.original": [
"/signewtestportal/upload"
],
"url.original.text": [
"/signewtestportal/upload"
],
"url.path": [
"/signewtestportal/upload"
],
"user_agent.device.name": [
"Other"
],
"user_agent.name": [
"Other"
],
"user_agent.original": [
"Dart/2.15 (dart:io)"
],
"user_agent.original.text": [
"Dart/2.15 (dart:io)"
],
"_id": "inwi_IwBPfAmZcwEY8vz",
"_index": ".ds-logs-nginx_ingress_controller.access-default-2024.01.11-000002",
"_score": null
}
Open a new topic with all the information...
Most likely you do not have enough resources... But exactly what unclear.
Be sure to include the specs for elasticsearch you have
How many agents
How many events per min or second etc
As much detail as you can provide
hi sir..@stephenb..the elastic agent running on kubernetes using the integration is collecting the previous dates logs also if i want to restrict the agent to collect the previous logs or to define certain time where i can do? kindly guide us