# Using container\_cpu\_usage\_total in Elasticsearch **URL:** \ **Category:** Metrics\ **Created:** [May 12, 2021, 7:36pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844 "2021-05-12T19:36:20Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\ **Post date:** [May 12, 2021, 7:36pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/1 "2021-05-12T19:36:20Z")
I'm collecting metrics from Kubernetes using Prometheus and sending them to ElasticSearch, with Metricbeat Prometheus module. ([Federation | Prometheus](https://prometheus.io/docs/prometheus/latest/federation/)). So, I have the following field in ElasticSearch Index: `prometheus.metrics.container_cpu_usage_total` . And I'm trying to use the following dashboard (TSVB Kibana) to get the usage percentage of CPU: [![enter image description here](https://us1.discourse-cdn.com/elastic/original/3X/e/e/eeba465620003832219de1089b8b5833a5bca5d1.png)](https://i.stack.imgur.com/RjzgM.png) I expected that using "Derivate" I have the same behavior that Prometheus Rate, but numbers returned in Dashboard no make sense for me, because is very different from `kubectl top pods` command. For example: In `kubectl top pods` I got 234m (millicores) in my pod, and in dashboard, I got numbers that vary between 80 and 90. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 13, 2021, 5:01pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/2 "2021-05-13T17:01:21Z")
Hi @Ronaldo_Lanhellas First you could use counter rate and that does the max, derivative positive only all in one step. Second you didn't pick a units 1s, 1m etc. Not sure what that defaults to. Give that a try see how it looks. ---
**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\ **Post date:** [May 13, 2021, 5:58pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/3 "2021-05-13T17:58:09Z")
> [@stephenb](#): > > counter rate I'm doing in this way: ![Captura de tela 2021-05-13 145732](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f1c25c934cc5c1dae4264241d64c958e0356bcc0.png) This is what you suggested? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 13, 2021, 6:31pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/4 "2021-05-13T18:31:06Z")
Yes, How does it Look? That assumes `prometheus.metrics.container_cpu_usage_total` is a monotonically increasing counter value. Also under panel options you should set `Interval` to \>= to your collection interval example if you are collecting every minute set to` >=1m` ![Screen Shot 2021-05-13 at 11.28.07 AM](https://us1.discourse-cdn.com/elastic/original/3X/d/7/d74b37570aeeb79d325422f70885ddc61c2ac5cf.png) Also you can format the units etc if you want ![Screen Shot 2021-05-13 at 11.28.35 AM](https://us1.discourse-cdn.com/elastic/original/3X/d/7/d73f4f0a786211a0f8dc558fc4521fa0441e3820.png) ---
**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\ **Post date:** [May 13, 2021, 7:44pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/5 "2021-05-13T19:44:01Z")
> [@stephenb](#): > > > = to yo I did everything that you posted but that value don't match with my `kubectl top nodes` command. This is the return from `kubectl top nodes`: ![Captura de tela 2021-05-13 163958](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5db0d176d2dd4452940fd5ed2579344fededaeab.png) As you can see I have about 12% CPU usage, and now in my Kibana I have the following: ![Captura de tela 2021-05-13 164258](https://us1.discourse-cdn.com/elastic/original/3X/4/2/4260b588352ba36005241b3e3c9d4a29fe53dbad.png) I got about 14% , sometime go to 19%, but continue 12% in `kubectl top nodes`. Really, I don't know what is correct, my kibana or my kubectl, I want to believe that my kubectl is correct. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 13, 2021, 8:10pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/6 "2021-05-13T20:10:34Z")
Hmm, Not sure; All Kibana is doing is taking the values from the prometheus exporter and doing the math to display... In my experience, It is often pretty hard to get exact comparisons from a CMD Line tool and and a metrics collections and visualizations. The Counter Rate in Kibana is pretty tested I use it all the time for network metrics and it is pretty solid (not to say there couldn't be an issue) You might need to do some deep reading on How `kubectl` is displaying data vs how the data is collected and reported by prometheus. I suspect there is nuances... [example](https://www.ibm.com/support/pages/kubectl-top-pods-and-docker-stats-show-different-memory-statistics) just some nuance on the memory collection (not cpu) but similar. Me? I would probably lean towards the prometheus collected metrics as that has such a wide user base, (assuming everything is configured correctly) and many folks monitor their K8s with prometheus collectors if there were issues I suspect they would be reported and fixed. I don't have a K8s cluster up and running right now so I can not compare. Perhaps someone else may chime in. ---
**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\ **Post date:** [May 13, 2021, 11:59pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/7 "2021-05-13T23:59:10Z")
> [@stephenb](#): > > K8s cluster up and running right now so I can no Thanks for your answer, do you think that I can use "Count Rate" as a percentage value ? Or I should use another kind of calculate to transform "Count Rate" to Percentage of usage ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 14, 2021, 1:53am UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/8 "2021-05-14T01:53:31Z")
Ok let's back up a bit... I am / was not reading carefully .... apologies _I was focused on showing you the correct way to calculate a rate._.. What we just calculated was the Rate of `container_cpu_usage_total` i.e the rate of CPU consumption (AND we should have SUM up the rates with a Series agg for all the Containers anyways which we did not do yet), not the percent CPU which I now understand what you wan So you want A) The Total CPU Percent for All Containers Per Node? B) The CPU Percent Per Container per Node? C) Both? So I think we need to do some re-thinking and yes you can do it, but it is going to take more work. (Of course this would most of this would be done for you automatically if you used metricbeat collection 🙂 ) I am not a prometheus expert looks like [something like these](https://stackoverflow.com/questions/40327062/how-to-calculate-containers-cpu-usage-in-kubernetes-with-prometheus-as-monitori) would be the types calculations we would need to do.. Pick A) or B) above and then find the other fields like total\_cores and the collection rate etc. Then perhaps I can help. If it is for all container we will need to sum the `container_cpu_usage_total` if for each container then will need to be broken down by container. Then it will looks something like this (This is not correct just and example) We will need to do a bucket script etc. and use the math from the Stack Overflow etc. ![Screen Shot 2021-05-13 at 6.05.43 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/d/ed0d98df57c3e3385efd0e2fcc48011b8814a4ea.png) ![Screen Shot 2021-05-13 at 6.05.49 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d307678963067c425178b12b278354598322a487.png) ---
**Author:** ![Ronaldo\_Lanhellas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronaldo_lanhellas/32/88020_2.png) [@Ronaldo\_Lanhellas](https://discuss.elastic.co/u/Ronaldo_Lanhellas)\ **Post date:** [May 31, 2021, 1:03pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/9 "2021-05-31T13:03:12Z")
Thanks for your detailed answer, I will follow your advice. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 28, 2021, 1:03pm UTC](https://discuss.elastic.co/t/using-container-cpu-usage-total-in-elasticsearch/272844/10 "2021-06-28T13:03:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.