# Kibana custom dashboard does not show Filebeat metrics values **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 28, 2022, 1:44am UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656 "2022-10-28T01:44:57Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [October 28, 2022, 1:44am UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/1 "2022-10-28T01:44:57Z")
Hi, **Summary** of the Issue I'm having: Custom Kibana dashboard I have created to show a few internal Filebeat metrics - does not show their values (just shows zeros for all fields I selected), even though I can see that the corresponding fields in the metrics events , in the .monitoring-beats-\* indices - do have non-zero values Now the details: After I finally got all Filebeat metrics successfully flowing into ES for monitoring ( see this post for the whole story and the final setup: [Filebeat monitoring metrics are "dropped" when a GEOIP pipeline is used - #24 by ppine7](https://discuss.elastic.co/t/filebeat-monitoring-metrics-are-dropped-when-a-geoip-pipeline-is-used/317063/24) ) - I now tried to show some of those metrics in Kibana dashboards... There is an already pre-build Kibana dashboard for Beats monitoring - which works good and I can see data there - but it only includes few main metrics. I wanted to add many more internal Filebeat metrics to help me troubleshoot some high load failure scenarios ... So, after reading about how to build a custom Dashboard with Filebeat metrics here: [Building Your Own Beat Dashboards | Beats Developer Guide [master] | Elastic](https://www.elastic.co/guide/en/beats/devguide/current/build-dashboards.html) I did the following: 1. verified I actually see those metrics I am interested in in the .monitoring-beat-\* indices... I am interested in all libbeat.output metrics - to see events delivery and failure statistics . Here is an example query I used to find metrics events that have one of those fields non-zero: ```auto GET .monitoring-beats-*/_search { "query": { "range": { "beats_stats.metrics.libbeat.output.events.total": { "gt": 100 } } } } ``` Since I ran a few load tests through Filebeat - I have ingested about 200K events into ES - there were bound to be metrics like that with non-zero values. And indeed, here is an example result with event like this: ```auto { "took": 601, "timed_out": false, "_shards": { "total": 3, "successful": 3, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 38, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": ".monitoring-beats-7-2022.10.27", "_id": "yWVVGoQBSGDuCsueuARX", "_score": 1, "_source": { "timestamp": "2022-10-27T16:45:35.951Z", "type": "beats_stats", "beats_stats": { "beat": { "uuid": "xxx555a", "type": "filebeat", "version": "8.4.3", "name": "mac-lt2-mpopova", "host": "mac-lt2-mpopova" }, "metrics": { "beat": { "memstats": { "gc_next": 43085040, "rss": 80461824, "memory_total": 2012057416, "memory_alloc": 29517288, "memory_sys": 70534152 }, "cpu": { "user": { "time": { "ms": 15264 }, "ticks": 15264 }, "system": { "ticks": 17221, "time": { "ms": 17221 } }, "total": { "value": 32485, "ticks": 32485, "time": { "ms": 32485 } } }, "runtime": { "goroutines": 74 }, "info": { "uptime": { "ms": 7800436 }, "ephemeral_id": "f70356f0-001c-4fd6-8437-d8f5d91bf54c", "name": "filebeat", "version": "8.4.3" } }, "system": { "cpu": { "cores": 16 }, "load": { "1": 3.1538, "5": 2.9277, "15": 2.5811, "norm": { "1": 0.1971, "5": 0.183, "15": 0.1613 } } }, "registrar": { "states": { "update": 0, "cleanup": 0, "current": 0 }, "writes": { "success": 0, "total": 0, "fail": 0 } }, "filebeat": { "harvester": { "started": 0, "closed": 0, "running": 0, "open_files": 0, "skipped": 0 }, "input": { "log": { "files": { "truncated": 0, "renamed": 0 } }, "netflow": { "flows": 0, "packets": { "dropped": 0, "received": 0 } } }, "events": { "active": 100, "added": 10021, "done": 9921 } }, "libbeat": { "config": { "reloads": 0, "module": { "starts": 0, "stops": 0, "running": 0 }, "scans": 0 }, "output": { "write": { "bytes": 15764310, "errors": 0 }, "read": { "errors": 2, "bytes": 145205 }, "type": "elasticsearch", "events": { "failed": 0, "dropped": 0, "duplicates": 0, "active": 50, "toomany": 0, "batches": 205, "total": 9971, "acked": 9921 } }, "pipeline": { "clients": 1, "events": { "published": 10021, "failed": 0, "dropped": 0, "retry": 1, "active": 100, "total": 10021, "filtered": 0 }, "queue": { "acked": 9921, "max_events": 4096 } } } }, "timestamp": "2022-10-27T16:45:35.951Z" }, "interval_ms": 10000, "cluster_uuid": "053kEhnTTqegfUyicL3J8g" } }, ``` especially interesting (to me) are the metrics I was looking for in the libbeat.output section: ```auto "libbeat": { "config": { "reloads": 0, "module": { "starts": 0, "stops": 0, "running": 0 }, "scans": 0 }, "output": { "write": { "bytes": 15764310, "errors": 0 }, "read": { "errors": 2, "bytes": 145205 }, "type": "elasticsearch", "events": { "failed": 0, "dropped": 0, "duplicates": 0, "active": 50, "toomany": 0, "batches": 205, "total": 9971, "acked": 9921 } } ``` 1. to visualize these libbeats metrics, I have created a new Data View (what used to be an index pattern in 7.x ES version): 2. and created a new Dashboard using this Data View. Here is an example of one visualization that is trying to show Suns of some of the "libbeat.output.xxx" metrics: 3. then I ran a few more load tests, making sure Fielbeat was busy sending data into ES. And still my new Dashboard does not show any data for those metrics - all values/sums are zeros: 4. Just to prove that there was indeed data flowing into ES from Filebeat - I have checked the "official" Beats Dashboard pre-packaged in ES - and it does who traffic going through the Filebeat: So , finally, the question: what did I do wrong in creating my custom Dashboard - that it does not show any metric values, even though those metrics / events do have non-zero values? Thank you! Marina ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 28, 2022, 6:00am UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/2 "2022-10-28T06:00:47Z")
Hi @ppine7 Lots of places to get tripped up here... 1st Use Lens, not the "legacy" visualizations 1A Super Important in your data view you need to use `timestamp` not `@timestamp` ![Screen Shot 2022-10-28 at 8.17.29 AM](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d3192a8cca0655a97524b369a721c902c2b19d13.png) 2nd Those values are monotonically increasing counters to visualize them as events per second you need to use a counter\_rate and be careful between `beats_state` and `beats_stats` 3rd you need to be careful with counters because they need to be applied Per Beat (counters / counter rates) are per entity and thus per type... because a counter\_rate takes the positive derivative of the max and the max could be different for each entity... 3rd See this Example `beats_stats.beat.type : "filebeat"` `counter_rate(max(beats_stats.metrics.libbeat.output.events.acked))` normalized to 1s Break down by the agent / beat host / id `beats_stats.beat.host` I only have 2 and they have equal rates ... and thus you get... ![Screen Shot 2022-10-27 at 11.03.16 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e2a8d86878a7d654ad81afea4fd15c085ef5ed1.jpeg) Close Up... Field: `beats_stats.metrics.libbeat.output.events.acked` ![Screen Shot 2022-10-27 at 11.05.20 PM](https://us1.discourse-cdn.com/elastic/original/3X/1/7/179705a66c3288adf7ee596eaf794ec2ba050044.jpeg) Which lines up with the monitoring ![Screen Shot 2022-10-28 at 8.22.34 AM](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68331f6c016c6fa2b266ef0dbdb5b6fe0cb51b3e.jpeg) ---
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [October 28, 2022, 4:10pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/3 "2022-10-28T16:10:27Z")
thank you, @stephenb ! I think I am using Lens - I was clicking on "Create vizualization" button when creating a new Dashboard - I think it is using Lens, if I am not mistaken... I was able to fix my dashboard following your suggestions. The issues I had to fix were: 1. use 'counter\_rate' instead of 'count' and 'Sum' 2. pick correct timestamp for X-axis: I was using 'beats\_state.timestamp' - but apparently I should have used just 'timestamp'. Still not sure why there are so many timestamps and why they differ.... but it works 3. bucketed by host - although in my case I have only one so it did not matter much so now I have this new vizualization: ![filebeat_dash_1028](https://us1.discourse-cdn.com/elastic/original/3X/7/8/782447bf9e63be04023fe8abbdc0d2234646db40.jpeg) if only I could get rid of the 'host' name as part of the legend metrics names.... and could see the actual names of metrics on the Y-axis .... right now they are very log and are created as a concat of "host\_name - metric\_name" and since the result is long - you can only see the first fiew characters of the name on the graph .... Any way to fix that? 🙂 Thank you for your help! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 28, 2022, 5:12pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/4 "2022-10-28T17:12:48Z")
> [@ppine7](#): > > if only I could get rid of the 'host' name as part of the legend metrics names.... and could see the actual names of metrics on the Y-axis .... Apologies... I am pretty confused ... For Y Axis either use the default or fill in how you please. The legend if you removed the host name how would you know which host the metric is for? I am confused seems like you need that... > [@ppine7](#): > > right now they are very log and are created as a concat of "host\_name - metric\_name" Can you show me that? I don't see that..I would like to see that You can set it to name + value .... You can set the leggend at the bottom... You could create a runtime field with a shortened host name... Lots of options... but I get it everyone want's there Viz... just so... ![Screen Shot 2022-10-28 at 10.10.37 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24c277870246b4c8c5d7b98ee5ff812a490060f4.png) ![Screen Shot 2022-10-28 at 10.09.12 AM](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5efad574a8343cff7f24f169559746b6f59823ae.png) I can see where if you put multiple Series on Same Graph it is hard to tell... not sure there is a fix for that right now Can you show me how you got it to be host + metricname? ---
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [October 28, 2022, 5:42pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/5 "2022-10-28T17:42:19Z")
Right, as you can see in this screenshot: ![filebeat_dash_yaxis](https://us1.discourse-cdn.com/elastic/original/3X/3/3/338afa3fdec2efb1540ed67f9f1e4dd299325104.jpeg) right now the legend for Y-axis shows cut off names of the metrics ... The full name is very long - and you can see it if you hover over the item only . What I'd like to see instead - is just the last part - like "event.acked" instead of the bucket+metricname, which is that long name with my full host name ... you can see the same in the full visualization screenshot too - all metric names are cut off and you can't tell which metric is for which line in the graph until you hover over it .... ![filebeat_dash_1028](https://us1.discourse-cdn.com/elastic/original/3X/7/8/782447bf9e63be04023fe8abbdc0d2234646db40.jpeg) thanks! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 28, 2022, 8:17pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/6 "2022-10-28T20:17:38Z")
And weird mine does not show the Axis Title mine shows the value when I turn in on You did not show me your detailed config / setup so I am not clear how you got that. But in the end probably not going to be able to magically fix this for you 🙂 You can also tell it not to truncate, or 2 lines or put at bottom... Or create a runtime field with just the short hostname... poke around.. the data is right... that is what is important... and besides you have the actual stack monitoring working now too right? ---
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [October 28, 2022, 8:47pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/7 "2022-10-28T20:47:22Z")
yes, you are right - the data is correct and it is all that matters 🙂 just in case something obviously wrong jumps out - I'm showing below how I defined the Y axis and bucketing - that resulted in that weird metrics naming in the legend: Vertical axis: multiple metrics (cut their number here for clarity): ![filebeat_axis_3](https://us1.discourse-cdn.com/elastic/original/3X/8/8/8879296750dd36d08be02072eb52125438213f11.jpeg) how one of the metrics is defined: ![filebeat_axis_2](https://us1.discourse-cdn.com/elastic/original/3X/7/a/7a7e97794c647c31bdfb2a7b38fb03052a76afdd.jpeg) how bucketing is defined: ![filebeat_axis_4](https://us1.discourse-cdn.com/elastic/original/3X/5/1/51c39c5d9faa596b07f8648984a4ce2124209cdb.jpeg) and this show how the metric labels in the legend become legible - as soon as I remove the bucketing by 'host' - which removes the host name form the metric name: ![filebeat_no_buckets_by_host](https://us1.discourse-cdn.com/elastic/original/3X/8/6/864cb45de15f21e6cb5e47b8ca9cee1a1161b01d.jpeg) Thanks!! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 28, 2022, 9:13pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/8 "2022-10-28T21:13:18Z")
Odd I never get host - axis name (metric) in the legend, perhaps we are on slightly different versions Sorry no magic.. . Options that I see - Turn Off Truncate Text - Multi-Line legend - Legend At Bottom - Create Short Name for Host using Runtime Field (which field are you using for host there may be the short name there) ![Screen Shot 2022-10-28 at 2.10.04 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/f/ef47fb6a48418573123d7e998c3a54e0727e5b06.png) ---
**Author:** ![ppine7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppine7/32/111560_2.png) [@ppine7](https://discuss.elastic.co/u/ppine7)\ **Post date:** [October 31, 2022, 3:24pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/9 "2022-10-31T15:24:54Z")
Thank you, @stephenb ! Just to close up on this: I was able to get a somewhat decent visualization by following your suggestions for Legends: - turned off truncation of the text - increased number of line to 2 - placed the legend at the bottom Also, when Filebeat is deployed on a GCP VM - the value in the beats\_stats.host becomes a GCP VM instance ID - which is much shorter than the actual host name, so it looks better. All in all - I call it a success 🙂 ![filebeat_monitoring_dashboard](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d88abf7608911453dc183f588b2afa2af6ecbd3.jpeg) Thank you!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 28, 2022, 5:25pm UTC](https://discuss.elastic.co/t/kibana-custom-dashboard-does-not-show-filebeat-metrics-values/317656/10 "2022-11-28T17:25:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.