# CPU usage metric via CAT API misbehaving after upgrading to 7.16.3 (running in LXD) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 17, 2022, 6:40pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620 "2022-01-17T18:40:04Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Post date:** [January 17, 2022, 6:40pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/1 "2022-01-17T18:40:04Z")
**Problem:** - In a 10 core CPU, 64GB RAM system, Centos 7.9 under LXD container virtualisation - Elasticsearch **reports wrong CPU usage** after upgrading to 7.16.3 - This issue wasn't present on 7.15.1 - We use this metric to track the health of the cluster but now it shows 100% CPU use most of the time - **This metric has been confirmed to not be true** using other OS CPU tools **Reproduce:** - Upgrade to Elasticsearch 7.16.3 - Give work to the cluster to increase the CPU usage - Obtain CPU usage via CAT API `curl -uuser:pass https://localhost:9200/_cat/nodes?v` - CAT API will report a sustained 100% CPU use ```auto # curl -uuser:pass https://localhost:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 10.x.x.x 6 98 100 21.44 23.53 24.91 dhimrsw - nodeXXXX 10.x.x.x 48 100 100 18.33 17.49 16.85 dhimrsw - nodeXXXX 10.x.x.x 32 99 100 10.52 12.45 13.57 dhimrsw - nodeXXXX 10.x.x.x 21 98 100 29.13 33.93 34.91 dhimrsw * nodeXXXX 10.x.x.x 35 100 100 16.02 17.13 16.59 dhimrsw - nodeXXXX 10.x.x.x 47 97 100 32.28 30.24 30.42 dhimrsw - nodeXXXX 10.x.x.x 41 95 100 8.22 7.02 7.23 dhimrsw - nodeXXXX 10.x.x.x 19 99 100 14.26 13.40 13.54 dhimrsw - nodeXXXX 10.x.x.x 51 95 100 30.86 28.44 27.42 dhimrsw - nodeXXXX 10.x.x.x 4 96 100 24.88 22.34 21.95 dhimrsw - nodeXXXX 10.x.x.x 23 96 100 14.34 15.02 16.20 dhimrsw - nodeXXXX 10.x.x.x 19 95 100 22.19 20.98 20.42 dhimrsw - nodeXXXX ``` Most of the time each node will report 100% CPU. We can see the effect on the metric after the patch and a rolling restart (the load in the cluster hasn't changed): ![cpu-usage-100-issue](https://us1.discourse-cdn.com/elastic/original/3X/9/7/971db1cbf15dea038ab8692e88119a086befcc78.png) Are we aware of this issue? Is it happening to anyone else? Thanks! Juan ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 17, 2022, 6:51pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/2 "2022-01-17T18:51:09Z")
Hi @juan.domenech What OS are you running on... is this Docker or Directly on the OS / VM? ---
**Author:** ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Post date:** [January 17, 2022, 6:55pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/3 "2022-01-17T18:55:12Z")
Hi! This is Centos 7.9 in a VM (10 cores and 64GB RAM per node and 12 nodes/VMs in the cluster). ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 17, 2022, 7:19pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/4 "2022-01-17T19:19:12Z")
Couple more questions Are you using the bundled JDK or your Own? Also curious what VM solution are you using... or is this AWS ec2 etc. ---
**Author:** ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Post date:** [January 17, 2022, 7:25pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/5 "2022-01-17T19:25:47Z")
No problem! - On-premise Hypervisors running LXD containers (we call them VM but I guess technically they are not) - Standard RPM, bundled JVM [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.3-x86\_64.rpm](https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.3-x86_64.rpm) Your question gave me an idea. We also have older Hypervisors running KVM (real VM). I'm going to test 7.16.3 there to see if the underlaying tech has something to do with this. I'll be back! ---
**Author:** ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Post date:** [January 17, 2022, 8:14pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/6 "2022-01-17T20:14:40Z")
I can confirm that in a VM running a traditional virtualisation layer (KVM) this issue is not present. Elasticsearch 7.16.3 reports CPU usage correctly. I'll update the original post accordingly. This does not explain why this metric broke between versions but it is an important clue. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 17, 2022, 8:33pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/7 "2022-01-17T20:33:39Z")
Could perhaps be related to a bug we opened against OpenJDK but may not explain why this affected when you upgraded. [https://bugs.openjdk.java.net/browse/JDK-8248215](https://bugs.openjdk.java.net/browse/JDK-8248215) ---
**Author:** ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Post date:** [January 19, 2022, 5:21pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/8 "2022-01-19T17:21:26Z")
I really don't know TBH. But seeing this changing between versions (without changes in our OS) makes me think about a code change. After a more detailed look to my graphs, I think that there is a bad calculation somewhere: ![single-vm-cpu](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9f8304f0efa7c8d372e206ba6e51b836ba07220c.jpeg) A node that in 7.15.1 was reporting around 7% CPU use, not reports around 70% with 7.16.3. If we factor in that this nodes is a 10 Core LXD container it looks like the CPU metric is multiplied by the number of cores and capped at 100. I see some recent changes on [OsProbe.java](https://github.com/pugnascotia/elasticsearch/commits/master/server/src/main/java/org/elasticsearch/monitor/os/OsProbe.java) in that area and I wonder if @rory.hunter could give us some directions 🙂 ---
**Author:** ![rory.hunter](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@rory.hunter](https://discuss.elastic.co/u/rory.hunter)\ **Post date:** [January 24, 2022, 10:20am UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/9 "2022-01-24T10:20:24Z")
My changes were to support cGroups v2. If the OS was using v2, then no metrics would have been available at all before those changes, which doesn't appear to be the case. The JVM bug seems relevant though. ---
**Author:** ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)\ **Post date:** [January 25, 2022, 8:43pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/10 "2022-01-25T20:43:01Z")
Yes it looks like a bug but not sure is the one mentioned earlier (Thanks @stephenb !) Between Elasticsearch versions Java went from 17 to 17.0.1 so it doesn't look like a big opportunity for that type of bug to get in there: ```auto [root@ ~]# rpm -qa|grep elasticsearch elasticsearch-7.15.1-1.x86_64 [root@ ~]# /usr/share/elasticsearch/jdk/bin/java -version openjdk version "17" 2021-09-14 OpenJDK Runtime Environment Temurin-17+35 (build 17+35) OpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing) ``` ```auto [root@ ~]# rpm -qa|grep elasticsearch elasticsearch-7.16.3-1.x86_64 [root@ ~]# /usr/share/elasticsearch/jdk/bin/java -version openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12) OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode, sharing) ``` Anyhow, let's wait a bit and see if someone else bumps into this (I'm afraid LXD containers are not very common). Thanks! Juan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 22, 2022, 8:43pm UTC](https://discuss.elastic.co/t/cpu-usage-metric-via-cat-api-misbehaving-after-upgrading-to-7-16-3-running-in-lxd/294620/11 "2022-02-22T20:43:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.