Hi,
Summary of the Issue I'm having:
Custom Kibana dashboard I have created to show a few internal Filebeat metrics - does not show their values (just shows zeros for all fields I selected), even though I can see that the corresponding fields in the metrics events , in the .monitoring-beats-* indices - do have non-zero values
Now the details:
After I finally got all Filebeat metrics successfully flowing into ES for monitoring ( see this post for the whole story and the final setup: Filebeat monitoring metrics are "dropped" when a GEOIP pipeline is used - #24 by ppine7 ) - I now tried to show some of those metrics in Kibana dashboards...
There is an already pre-build Kibana dashboard for Beats monitoring - which works good and I can see data there - but it only includes few main metrics. I wanted to add many more internal Filebeat metrics to help me troubleshoot some high load failure scenarios ...
So, after reading about how to build a custom Dashboard with Filebeat metrics here: Building Your Own Beat Dashboards | Beats Developer Guide [master] | Elastic I did the following:
- verified I actually see those metrics I am interested in in the .monitoring-beat-* indices... I am interested in all libbeat.output metrics - to see events delivery and failure statistics . Here is an example query I used to find metrics events that have one of those fields non-zero:
GET .monitoring-beats-*/_search
{
"query": {
"range": {
"beats_stats.metrics.libbeat.output.events.total": {
"gt": 100
}
}
}
}
Since I ran a few load tests through Filebeat - I have ingested about 200K events into ES - there were bound to be metrics like that with non-zero values. And indeed, here is an example result with event like this:
{
"took": 601,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 38,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": ".monitoring-beats-7-2022.10.27",
"_id": "yWVVGoQBSGDuCsueuARX",
"_score": 1,
"_source": {
"timestamp": "2022-10-27T16:45:35.951Z",
"type": "beats_stats",
"beats_stats": {
"beat": {
"uuid": "xxx555a",
"type": "filebeat",
"version": "8.4.3",
"name": "mac-lt2-mpopova",
"host": "mac-lt2-mpopova"
},
"metrics": {
"beat": {
"memstats": {
"gc_next": 43085040,
"rss": 80461824,
"memory_total": 2012057416,
"memory_alloc": 29517288,
"memory_sys": 70534152
},
"cpu": {
"user": {
"time": {
"ms": 15264
},
"ticks": 15264
},
"system": {
"ticks": 17221,
"time": {
"ms": 17221
}
},
"total": {
"value": 32485,
"ticks": 32485,
"time": {
"ms": 32485
}
}
},
"runtime": {
"goroutines": 74
},
"info": {
"uptime": {
"ms": 7800436
},
"ephemeral_id": "f70356f0-001c-4fd6-8437-d8f5d91bf54c",
"name": "filebeat",
"version": "8.4.3"
}
},
"system": {
"cpu": {
"cores": 16
},
"load": {
"1": 3.1538,
"5": 2.9277,
"15": 2.5811,
"norm": {
"1": 0.1971,
"5": 0.183,
"15": 0.1613
}
}
},
"registrar": {
"states": {
"update": 0,
"cleanup": 0,
"current": 0
},
"writes": {
"success": 0,
"total": 0,
"fail": 0
}
},
"filebeat": {
"harvester": {
"started": 0,
"closed": 0,
"running": 0,
"open_files": 0,
"skipped": 0
},
"input": {
"log": {
"files": {
"truncated": 0,
"renamed": 0
}
},
"netflow": {
"flows": 0,
"packets": {
"dropped": 0,
"received": 0
}
}
},
"events": {
"active": 100,
"added": 10021,
"done": 9921
}
},
"libbeat": {
"config": {
"reloads": 0,
"module": {
"starts": 0,
"stops": 0,
"running": 0
},
"scans": 0
},
"output": {
"write": {
"bytes": 15764310,
"errors": 0
},
"read": {
"errors": 2,
"bytes": 145205
},
"type": "elasticsearch",
"events": {
"failed": 0,
"dropped": 0,
"duplicates": 0,
"active": 50,
"toomany": 0,
"batches": 205,
"total": 9971,
"acked": 9921
}
},
"pipeline": {
"clients": 1,
"events": {
"published": 10021,
"failed": 0,
"dropped": 0,
"retry": 1,
"active": 100,
"total": 10021,
"filtered": 0
},
"queue": {
"acked": 9921,
"max_events": 4096
}
}
}
},
"timestamp": "2022-10-27T16:45:35.951Z"
},
"interval_ms": 10000,
"cluster_uuid": "053kEhnTTqegfUyicL3J8g"
}
},
especially interesting (to me) are the metrics I was looking for in the libbeat.output section:
"libbeat": {
"config": {
"reloads": 0,
"module": {
"starts": 0,
"stops": 0,
"running": 0
},
"scans": 0
},
"output": {
"write": {
"bytes": 15764310,
"errors": 0
},
"read": {
"errors": 2,
"bytes": 145205
},
"type": "elasticsearch",
"events": {
"failed": 0,
"dropped": 0,
"duplicates": 0,
"active": 50,
"toomany": 0,
"batches": 205,
"total": 9971,
"acked": 9921
}
}
-
to visualize these libbeats metrics, I have created a new Data View (what used to be an index pattern in 7.x ES version):
-
and created a new Dashboard using this Data View. Here is an example of one visualization that is trying to show Suns of some of the "libbeat.output.xxx" metrics:
-
then I ran a few more load tests, making sure Fielbeat was busy sending data into ES. And still my new Dashboard does not show any data for those metrics - all values/sums are zeros:
-
Just to prove that there was indeed data flowing into ES from Filebeat - I have checked the "official" Beats Dashboard pre-packaged in ES - and it does who traffic going through the Filebeat:
So , finally, the question: what did I do wrong in creating my custom Dashboard - that it does not show any metric values, even though those metrics / events do have non-zero values?
Thank you!
Marina
