Convert message into eps data and create a visualization

Elastic Stack Elasticsearch
1

Note - I do have a ticket in with support but I felt that maybe this would be useful to the community as well so I will update things as we move along.

What were doing:
Shipping from 9 DLC's running in Openshift with Filebeat to our Elastic deployment. The Filebeat is sending data to an ingest pipeline called eps-data that has the following setup. The ingest pipeline config is designed to utilize the grok and convert processors to pull the eps values out and then change the string to a float to be utilized for visualization.

Sample Log
"message": [
"[0] dlc_eps: [1689790390.673000000, {}, {"eps5sec"=>5164.600000, "eps10sec"=>"5088.60", "eps15sec"=>"5153.60", "eps30sec"=>"5497.33", "eps60sec"=>"5301.27", "eps300sec"=>"5365.53", "eps900sec"=>"5365.53", "peak60sec"=>"6490.20", "peak"=>"9196.60", "throttles5sec"=>"0.00", "throttles60sec"=>"0", "throttlestotal"=>"0", "threshold"=>"1.0E8"}]"

DLC Field
"kubernetes.pod.name": [
"dlc-9"
]

Processors
[ { "grok": { "field": "message", "patterns": [ "(%{GREEDYDATA:drop_start} \{\"eps5sec\"=>%{NUMBER:5_sec_eps}, \"eps10sec\"=>\"%{NUMBER:10_sec_eps}\", \"eps15sec\"=>\"%{NUMBER:15_sec_eps}\", \"eps30sec\"=>\"%{NUMBER:30_sec_eps}\", \"eps60sec\"=>\"%{NUMBER:60_sec_eps}\", \"eps300sec\"=>\"%{NUMBER:300_sec_eps}\", \"eps900sec\"=>\"%{NUMBER:900_sec_eps}\", \"peak60sec\"=>\"%{NUMBER:60_sec_peak}\", \"peak\"=>\"%{NUMBER:peak_sec}\", \"throttles5sec\"=>\"%{NUMBER:5_sec_throttles}\", \"throttles60sec\"=>\"%{NUMBER:60_sec_throttles}\", \"throttlestotal\"=>\"%{NUMBER:total_throttles}\", %{GREEDYDATA:drop_end}\"}])" ] } }, { "convert": { "field": "5_sec_eps", "type": "integer" } }, { "convert": { "field": "10_sec_eps", "type": "float" } }, { "convert": { "field": "15_sec_eps", "type": "float" } }, { "convert": { "field": "30_sec_eps", "type": "float" } }, { "convert": { "field": "60_sec_eps", "type": "float" } }, { "convert": { "field": "300_sec_eps", "type": "float" } }, { "convert": { "field": "900_sec_eps", "type": "float" } } ]
Failure Processors
[ { "append": { "field": "error.message", "value": [ "{{{_ingest.on_failure_message}}}" ] } } ]

What were trying to do
Plot the 5, 10, 15, etc, EPS values for each DLC that we get data from as a visualization.

Where were stuck
The issue, I think, is that even though the values are being converted from the string to a float it might be stored as a keyword instead of a number. Or we may even have an issue where the fields are being stored as a nested array which makes it difficult to create the visualization.

2

Yup , most likely ... can you share the mapping? Do you know how to do that?

You will need to create a template (aka mapping ) with the correct types

Can you share a sample document?

And btw when you say plot lines, viz are always aggregations in elastic, it will always be average, sum, max etc... in general kibana does not plot single values of course the average of a single value is the single value :wink:

3

Any recommendations on how to post the mappings and a sample document without hitting this error? Remove the blank space seems like the way to go so I'm working on figuring that out.

image
image817×192 7.71 KB

Edit:
An error occurred: Body is limited to 35000 characters; you entered
Possible solutions: Split the responses into 2 responses or utilize something like Notepad++ and I figured it out, in Notepad I just used Edit --> Blank Operations --> Trim Trailing and Trim Leading Space may help get rid of the extra characters.

4

@stephenb I think I've got everything requested. Just had to scrub some fields this morning.
Mapping:

#GET ocp-prod-its-prod-siem-infrastructure-dlc/_mapping
{
".dlc-infrastructure" : {
"mappings" : {
"_data_stream_timestamp" : {
"enabled" : true
},
"properties" : {
"10_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"15_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"300_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"30_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"5_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"5_sec_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_peak" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"900_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"@timestamp" : {
"type" : "date"
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"container" : {
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"image" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"runtime" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"drop_end" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"drop_start" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"error" : {
"properties" : {
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"host" : {
"properties" : {
"architecture" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"containerized" : {
"type" : "boolean"
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ip" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"mac" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"os" : {
"properties" : {
"codename" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"family" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kernel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"platform" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"kubernetes" : {
"properties" : {
"container" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"labels" : {
"properties" : {
"app_kubernetes_io/instance" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"app_kubernetes_io/name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"controller-revision-hash" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"statefulset_kubernetes_io/pod-name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"namespace" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"namespace_labels" : {
"properties" : {
"kubernetes_io/metadata_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"openshift-pipelines_tekton_dev/namespace-reconcile-version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"namespace_uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node" : {
"properties" : {
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"labels" : {
"properties" : {
"beta_kubernetes_io/arch" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"beta_kubernetes_io/os" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/arch" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/os" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node-role_kubernetes_io/worker" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"nodeID" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node_openshift_io/os_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"storage" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"twistlocknode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"pod" : {
"properties" : {
"ip" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"statefulset" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"log" : {
"properties" : {
"file" : {
"properties" : {
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"offset" : {
"type" : "long"
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"peak_sec" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"stream" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"total_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
},
".dlc-infrastructure" : {
"mappings" : {
"_data_stream_timestamp" : {
"enabled" : true
},
"properties" : {
"10_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"15_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"300_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"30_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"5_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"5_sec_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_peak" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"900_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"@timestamp" : {
"type" : "date"
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"container" : {
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"image" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"runtime" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"drop_end" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"drop_start" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"error" : {
"properties" : {
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"host" : {
"properties" : {
"architecture" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"containerized" : {
"type" : "boolean"
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ip" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"mac" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"os" : {
"properties" : {
"codename" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"family" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kernel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"platform" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"kubernetes" : {
"properties" : {
"container" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"labels" : {
"properties" : {
"app_kubernetes_io/instance" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"app_kubernetes_io/name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"controller-revision-hash" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"statefulset_kubernetes_io/pod-name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"namespace" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"namespace_labels" : {
"properties" : {
"kubernetes_io/metadata_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"openshift-pipelines_tekton_dev/namespace-reconcile-version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"namespace_uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node" : {
"properties" : {
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"labels" : {
"properties" : {
"beta_kubernetes_io/arch" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"beta_kubernetes_io/os" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/arch" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/os" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node-role_kubernetes_io/worker" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"nodeID" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node_openshift_io/os_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"storage" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"twistlocknode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"pod" : {
"properties" : {
"ip" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"statefulset" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"log" : {
"properties" : {
"file" : {
"properties" : {
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"offset" : {
"type" : "long"
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"peak_sec" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"stream" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"total_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
},
"shrink-fway-.dlc-infrastructure" : {
"mappings" : {
"_data_stream_timestamp" : {
"enabled" : true
},
"properties" : {
"10_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"15_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"300_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"30_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"5_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"5_sec_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_peak" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"60_sec_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"900_sec_eps" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"@timestamp" : {
"type" : "date"
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"container" : {
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"image" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"runtime" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"drop_end" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"drop_start" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"error" : {
"properties" : {
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"host" : {
"properties" : {
"architecture" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"containerized" : {
"type" : "boolean"
},
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ip" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"mac" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"os" : {
"properties" : {
"codename" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"family" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kernel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"platform" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"kubernetes" : {
"properties" : {
"container" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"labels" : {
"properties" : {
"app_kubernetes_io/instance" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"app_kubernetes_io/name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"controller-revision-hash" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"statefulset_kubernetes_io/pod-name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"namespace" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"namespace_labels" : {
"properties" : {
"kubernetes_io/metadata_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"openshift-pipelines_tekton_dev/namespace-reconcile-version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"namespace_uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node" : {
"properties" : {
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"labels" : {
"properties" : {
"beta_kubernetes_io/arch" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"beta_kubernetes_io/os" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/arch" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"kubernetes_io/os" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node-role_kubernetes_io/worker" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"nodeID" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"node_openshift_io/os_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"storage" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"twistlocknode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"pod" : {
"properties" : {
"ip" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"statefulset" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"log" : {
"properties" : {
"file" : {
"properties" : {
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"offset" : {
"type" : "long"
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"peak_sec" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"stream" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"total_throttles" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}

Example document:

{
"_index": ".dlc-infrastructure",
"_id": "yK-HeIkBTnfFfnI4ujSC",
"_version": 1,
"_score": 1,
"_ignored": [
"message.keyword"
],
"_source": {
"@timestamp": "2023-07-21T12:58:21.352Z",
"log": {
"offset": 6283114,
"file": {
"path": "/var/log/containers/infrastructure-dlc_dlc-1234.log"
}
},
"stream": "stdout",
"message": "[0] dlc_eps: [1689944058.764000000, {}, {\"eps5sec\"=>5488.200000, \"eps10sec\"=>\"4803.40\", \"eps15sec\"=>\"5836.73\", \"eps30sec\"=>\"5584.90\", \"eps60sec\"=>\"5677.70\", \"eps300sec\"=>\"5370.90\", \"eps900sec\"=>\"5370.90\", \"peak60sec\"=>\"7903.40\", \"peak\"=>\"8901.40\", \"throttles5sec\"=>\"0.00\", \"throttles60sec\"=>\"0\", \"throttlestotal\"=>\"0\", \"threshold\"=>\"1.0E8\"}]",
"input": {
"type": "container"
},
"kubernetes": {
"namespace_uid": "123456",
"namespace_labels": {
"kubernetes_io/metadata_name": "dlc-infrastructure",
"openshift-pipelines_tekton_dev/namespace-reconcile-version": "v1.6.4"
},
"pod": {
"uid": "123456",
"ip": "1.1.1.1",
"name": "dlc"
},
"namespace": "dlc-infrastructure",
"statefulset": {
"name": "dlc"
},
"labels": {
"app_kubernetes_io/instance": "dlc",
"app_kubernetes_io/name": "dlc",
"controller-revision-hash": "dlc-123456",
"statefulset_kubernetes_io/pod-name": "dlc"
},
"container": {
"name": "dlc"
},
"node": {
"labels": {
"node-role_kubernetes_io/worker": "",
"nodeID": "worker.gov",
"node_openshift_io/os_id": "rhcos",
"beta_kubernetes_io/arch": "amd64",
"beta_kubernetes_io/os": "linux",
"kubernetes_io/arch": "amd64",
"twistlocknode": "true",
"storage": "storagenode",
"kubernetes_io/os": "linux",
"kubernetes_io/hostname": "worker.gov"
},
"hostname": "worker.gov",
"name": "worker.gov",
"uid": "123456"
}
},
"host": {
"hostname": "worker.gov",
"architecture": "x86_64",
"name": "worker.gov",
"os": {
"family": "debian",
"name": "Ubuntu",
"kernel": "4.18.0-305.86.2.el8_4.x86_64",
"codename": "focal",
"type": "linux",
"platform": "ubuntu",
"version": "20.04.3 LTS (Focal Fossa)"
},
"containerized": true,
"ip": [
"1.1.1.1"
],
"mac": [
"00:00:00:00"
]
},
"container": {
"runtime": "cri-o",
"image": {
"name": "registry"
},
"id": "123456"
},
"ecs": {
"version": "1.12.0"
},
"agent": {
"id": "123456",
"name": "worker.gov",
"type": "filebeat",
"version": "7.17.0",
"hostname": "worker.gov",
"ephemeral_id": "123456"
}
},
"fields": {
"agent.version.keyword": [
"7.17.0"
],
"kubernetes.node.uid": [
"123456"
],
"kubernetes.namespace_uid.keyword": [
"123456"
],
"host.name.keyword": [
"worker.gov"
],
"kubernetes.namespace_uid": [
"123456"
],
"host.hostname": [
"worker.gov"
],
"host.mac": [
"00:00:00:00"
],
"kubernetes.node.labels.kubernetes_io/os": [
"linux"
],
"container.id": [
"123456"
],
"container.image.name": [
"registry"
],
"kubernetes.container.name.keyword": [
"dlc"
],
"host.os.version": [
"20.04.3 LTS (Focal Fossa)"
],
"kubernetes.node.labels.beta_kubernetes_io/os": [
"linux"
],
"kubernetes.pod.name.keyword": [
"dlc"
],
"kubernetes.namespace_labels.openshift-pipelines_tekton_dev/namespace-reconcile-version.keyword": [
"v1.6.4"
],
"agent.name": [
"worker.gov"
],
"kubernetes.labels.app_kubernetes_io/name": [
"dlc"
],
"kubernetes.statefulset.name.keyword": [
"dlc"
],
"host.os.type": [
"linux"
],
"kubernetes.node.labels.kubernetes_io/hostname.keyword": [
"worker.gov"
],
"agent.id.keyword": [
"123456"
],
"input.type": [
"container"
],
"kubernetes.node.uid.keyword": [
"123456"
],
"agent.hostname": [
"worker.gov"
],
"kubernetes.labels.statefulset_kubernetes_io/pod-name.keyword": [
"dlc"
],
"host.architecture": [
"x86_64"
],
"agent.id": [
"123456"
],
"host.containerized": [
true
],
"kubernetes.node.labels.twistlocknode": [
"true"
],
"input.type.keyword": [
"container"
],
"stream.keyword": [
"stdout"
],
"kubernetes.node.hostname": [
"worker.gov"
],
"kubernetes.node.labels.storage.keyword": [
"storagenode"
],
"kubernetes.node.hostname.keyword": [
"worker.gov"
],
"host.ip": [
"1.1.1.1"
],
"agent.type": [
"filebeat"
],
"host.os.kernel.keyword": [
"4.18.0-305.86.2.el8_4.x86_64"
],
"stream": [
"stdout"
],
"container.image.name.keyword": [
"registry"
],
"agent.type.keyword": [
"filebeat"
],
"kubernetes.pod.ip": [
"1.1.1.1"
],
"agent.ephemeral_id.keyword": [
"123456"
],
"kubernetes.node.labels.node_openshift_io/os_id.keyword": [
"rhcos"
],
"kubernetes.container.name": [
"dlc"
],
"agent.name.keyword": [
"worker.gov"
],
"kubernetes.node.labels.beta_kubernetes_io/arch.keyword": [
"amd64"
],
"host.os.codename": [
"focal"
],
"kubernetes.labels.app_kubernetes_io/instance.keyword": [
"dlc"
],
"kubernetes.node.labels.beta_kubernetes_io/arch": [
"amd64"
],
"@timestamp": [
"2023-07-21T12:58:21.352Z"
],
"kubernetes.pod.uid.keyword": [
"123456"
],
"container.runtime.keyword": [
"cri-o"
],
"host.os.platform": [
"ubuntu"
],
"log.file.path": [
"/var/log/containers/"
],
"agent.ephemeral_id": [
"123456"
],
"kubernetes.node.labels.kubernetes_io/arch": [
"amd64"
],
"host.architecture.keyword": [
"x86_64"
],
"kubernetes.namespace_labels.openshift-pipelines_tekton_dev/namespace-reconcile-version": [
"v1.6.4"
],
"agent.hostname.keyword": [
"worker.gov"
],
"kubernetes.node.labels.kubernetes_io/os.keyword": [
"linux"
],
"ecs.version.keyword": [
"1.12.0"
],
"host.ip.keyword": [
"1.1.1.1"
],
"kubernetes.node.labels.twistlocknode.keyword": [
"true"
],
"kubernetes.node.labels.node_openshift_io/os_id": [
"rhcos"
],
"kubernetes.namespace": [
"dlc-infrastructure"
],
"host.os.name": [
"Ubuntu"
],
"host.name": [
"worker.gov"
],
"kubernetes.labels.statefulset_kubernetes_io/pod-name": [
"dlc"
],
"kubernetes.node.labels.node-role_kubernetes_io/worker.keyword": [
""
],
"host.os.version.keyword": [
"20.04.3 LTS (Focal Fossa)"
],
"kubernetes.labels.app_kubernetes_io/instance": [
"dlc"
],
"kubernetes.node.labels.storage": [
"storagenode"
],
"log.offset": [
6283114
],
"container.runtime": [
"cri-o"
],
"ecs.version": [
"1.12.0"
],
"host.hostname.keyword": [
"worker.gov"
],
"agent.version": [
"7.17.0"
],
"kubernetes.namespace.keyword": [
"dlc-infrastructure"
],
"host.os.family": [
"debian"
],
"kubernetes.node.name": [
"worker.gov"
],
"kubernetes.node.name.keyword": [
"worker.gov"
],
"kubernetes.pod.uid": [
"123456"
],
"kubernetes.labels.controller-revision-hash.keyword": [
"dlc-123456"
],
"kubernetes.node.labels.nodeID.keyword": [
"worker.gov"
],
"host.os.kernel": [
"4.18.0-305.86.2.el8_4.x86_64"
],
"kubernetes.node.labels.kubernetes_io/arch.keyword": [
"amd64"
],
"host.os.name.keyword": [
"Ubuntu"
],
"kubernetes.pod.name": [
"dlc"
],
"kubernetes.node.labels.nodeID": [
"worker.gov"
],
"log.file.path.keyword": [
"/var/log/containers/.log"
],
"kubernetes.node.labels.node-role_kubernetes_io/worker": [
""
],
"host.os.codename.keyword": [
"focal"
],
"host.mac.keyword": [
"00:00:00:00",
],
"kubernetes.namespace_labels.kubernetes_io/metadata_name": [
"dlc-infrastructure"
],
"message": [
"[0] dlc_eps: [1689944058.764000000, {}, {\"eps5sec\"=>5488.200000, \"eps10sec\"=>\"4803.40\", \"eps15sec\"=>\"5836.73\", \"eps30sec\"=>\"5584.90\", \"eps60sec\"=>\"5677.70\", \"eps300sec\"=>\"5370.90\", \"eps900sec\"=>\"5370.90\", \"peak60sec\"=>\"7903.40\", \"peak\"=>\"8901.40\", \"throttles5sec\"=>\"0.00\", \"throttles60sec\"=>\"0\", \"throttlestotal\"=>\"0\", \"threshold\"=>\"1.0E8\"}]"
],
"kubernetes.node.labels.kubernetes_io/hostname": [
"worker.gov"
],
"host.os.family.keyword": [
"debian"
],
"kubernetes.statefulset.name": [
"dlc"
],
"host.os.type.keyword": [
"linux"
],
"host.os.platform.keyword": [
"ubuntu"
],
"kubernetes.labels.app_kubernetes_io/name.keyword": [
"dlc"
],
"kubernetes.namespace_labels.kubernetes_io/metadata_name.keyword": [
"dlc-infrastructure"
],
"kubernetes.labels.controller-revision-hash": [
"dlc-123456"
],
"container.id.keyword": [
"123456"
],
"kubernetes.node.labels.beta_kubernetes_io/os.keyword": [
"linux"
],
"kubernetes.pod.ip.keyword": [
"1.1.1.1"
]
},
"ignored_field_values": {
"message.keyword": [
"[0] dlc_eps: [1689944058.764000000, {}, {\"eps5sec\"=>5488.200000, \"eps10sec\"=>\"4803.40\", \"eps15sec\"=>\"5836.73\", \"eps30sec\"=>\"5584.90\", \"eps60sec\"=>\"5677.70\", \"eps300sec\"=>\"5370.90\", \"eps900sec\"=>\"5370.90\", \"peak60sec\"=>\"7903.40\", \"peak\"=>\"8901.40\", \"throttles5sec\"=>\"0.00\", \"throttles60sec\"=>\"0\", \"throttlestotal\"=>\"0\", \"threshold\"=>\"1.0E8\"}]"
]
}
}
5

Yes that is the problem you need to create a mapping before you ingest (an index template) and set those to a float type... I do not see those in your sample document.

Elasticsearch should have guested the right type but I suspect the sample doce had those values in quotes so it interpreted them as strings

6

Looking much better! Thanks for the guidance.

image
image967×806 45.9 KB

2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.