# High system CPU usage with Elasticsearch on Kubernetes (using ECK) using ES v7.10.0+ (but not in v7.6.1) **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [December 3, 2021, 2:21pm UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904 "2021-12-03T14:21:13Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![YannV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yannv/32/98122_2.png) [@YannV](https://discuss.elastic.co/u/YannV)\ **Post date:** [December 3, 2021, 2:21pm UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/1 "2021-12-03T14:21:13Z")
**Elasticsearch version** : 7.10.0 & 7.14.2 (but not reproduced in ES v7.6.1) **Plugins installed** : None **JVM version** : Bundled in official Elasticsearch docker image **OS version** : Reproduced on CentOS 7.9 with kernel 3.10.0 and 5.4.155 ; as well as on Ubuntu 20.04 with kernel 5.4.0 **OS version** : Kubernetes 1.19.x & Kubernetes v1.21.5 **Description of the problem including expected versus actual behavior** : Running an **Elasticsearch 7.10.0+ cluster on Kubernetes** (reproduced on two different K8s distributions) using ECK 1.8.0. While ingesting documents in Elaticsearch and running performance tests, we noticed a high (and unusual) system CPU usage (between 20 and 30% while we are CPU bound, basically user cpu is around 60% and with very little IO-Wait). We have reproduced this high system CPU usage with ES v7.10.0, as well as with ES v7.14.2 to check if it was still present in newer releases. It does not seem to be OS specific nor from the OS kernel versions as described above. Similarly for Kubernetes version used, and storage layer (CSI) does not matter either. We have been using statically/local Persistent volumes and OpenEBS, and in all cases the high system CPU usage issue was present with Elasticsearch v7.10.0+. _However, if we have Elasticsearch on the same topology and same hardware but only running on Docker (so without Kubernetes), this high system CPU usage is not there, so it is not just a containerized ES issue._ Moreover we did not noticed this over a year ago, but we were using the Elasticsearch version released from back then, being v7.6.1. We have just tested it again with this old ES version and sure enough we do not reproduce the high system cpu usage under the exact same condition (same data, same hardware, same ES topology, same K8s cluster, same ECK, just changing the ES version from the manifest). Therefore, we believe that something changed in Elasticsearch between v7.6.1 and v7.10.0 that causes this high system CPU usage when running on top of Kubernetes. **Steps to reproduce** : 1. Deploy an **ES v7.10.0+** cluster on Kubernetes with ECK and start ingesting enough documents to reach a CPU bound state (high CPU usage on Elasticsearch Data node). We used _ESRally_ for this with eventdata track. Using one shard per vCores allocated of the ES cluster (ES Data pods). The number of ES Data nodes does not matter we have reproduce it on 2 nodes and 75 nodes clusters. Having dedicated master nodes or without ES master nodes does not matter either we have tested these scenarios. 2. Check the different CPU usage with something like _dstat -lrvn_ for instance and focus on system CPU value. 3. You should see going above 20%. 4. Now if you do the same test with **ES v7.6.1** then the system cpu usage will be much lower. This high system CPU usage (20-30%) prevents us to reach our previous ingestion rate we had last year. ![ES-High-system-CPU-usage-on-K8S-v7.10.0-vs-v7.6.1](https://us1.discourse-cdn.com/elastic/original/3X/4/2/4200667a0a48a8661724dcf7dcc968d025d4c09a.png) ![ES-High-system-CPU-usage-on-K8S-DifferentOS-DifferentK8sDistro](https://us1.discourse-cdn.com/elastic/original/3X/2/5/25ba927e9457a8f7f8fd8664584f023c1c9fda2d.png) ---
**Author:** ![YannV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yannv/32/98122_2.png) [@YannV](https://discuss.elastic.co/u/YannV)\ **Post date:** [December 3, 2021, 6:05pm UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/2 "2021-12-03T18:05:58Z")
We have performed more tests on different Elasticsearch versions to narrow this down. Our conclusion is that this high system CPU usage in Elasticsearch during data ingestion started in Elasticsearch v7.9.0 ![ES-High-system-CPU-usage-on-K8S-v7.8.0-CPU](https://us1.discourse-cdn.com/elastic/original/3X/0/2/02d6c53c56a62844f45da0f9447d809b0016ec66.png) ![ES-High-system-CPU-usage-on-K8S-v7.9.0-CPU](https://us1.discourse-cdn.com/elastic/original/3X/e/5/e577e95f9d3dfb43e73129140ac1e8d4f2340d1d.png) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 6, 2021, 1:47am UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/3 "2021-12-06T01:47:40Z")
Welcome to our community! 😃 What sort of data are you ingesting? How are you running these tests? ---
**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\ **Post date:** [December 7, 2021, 12:51am UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/4 "2021-12-07T00:51:37Z")
Heya @YannV - thanks for creating the discuss post ([directed from this GitHub issue](https://github.com/elastic/elasticsearch/issues/81227)). Along with Mark's request for the format of your data and how you are running your tests (i.e. your methodology and tooling), can you please share: - The manifests you're using to deploy your clusters with - The output of the [Hot Threads API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cluster-nodes-hot-threads.html) to see what ES is actually spending CPU time on during the benchmark. Bonus points if you can get comparisons between both 7.6 and 7.10, that'd be even better. ---
**Author:** ![YannV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yannv/32/98122_2.png) [@YannV](https://discuss.elastic.co/u/YannV)\ **Post date:** [December 9, 2021, 6:39pm UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/5 "2021-12-09T18:39:20Z")
Just to let you know, if it can help someone else, we have find the root cause of the problem. We were using _ **node.store.allow\_mmap: false** _ in the configuration of our Elasticsearch cluster deployment file and this was the reason behind the high system CPU. This was clear in the [documentation](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-virtual-memory.html#k8s-virtual-memory): > _For production workloads, it is strongly recommended to increase the kernel setting vm.max\_map\_count to 262144 and leave node.store.allow\_mmap unset._ ---
**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\ **Post date:** [December 9, 2021, 10:20pm UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/6 "2021-12-09T22:20:19Z")
Thanks for letting us all know! This was my primary suspicion, as without setting `node.store.allow_mmap: true`, Elasticsearch falls back to using [`niofs`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-store.html). `niofs` uses the `read()` syscall to access the underlying segment files, which incurs a context switching overhead, which is what you see reflected in the increase in CPU time spent in the kernel (system). Starting from 7.x, the default index store type is `hybridfs` , which chooses different strategies to read Lucene files based on the read access pattern (random or sequential) in order to optimise performance, i.e. `niofs` using the `read()` syscall, or `mmapfs` using the `mmap()` syscall. By not setting `node.store.allow_mmap: true`, you're forcing Elasticsearch to open _all_ files via `niofs`. Hope this helps explain it a little more 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 6, 2022, 10:20pm UTC](https://discuss.elastic.co/t/high-system-cpu-usage-with-elasticsearch-on-kubernetes-using-eck-using-es-v7-10-0-but-not-in-v7-6-1/290904/7 "2022-01-06T22:20:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.