# 100% cpu system time used on hdd data node

**URL:** <https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532>\
**Category:** Elasticsearch\
**Created:** [March 17, 2021, 2:43pm UTC](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532 "2021-03-17T14:43:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\
**Post date:** [March 17, 2021, 2:43pm UTC](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532/1 "2021-03-17T14:43:19Z")

</div>

The ES version is 7.11.1  
We use 16C64G vm which has 4 physical hdd disks(striped lvm volume) as warm data node.

The issue is that the cpu system time of a random vm offen suddenly rises up to 100%, and then the vm keeps hanging until it leaves the cluster.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac334261c9d502e1479b83fb0c0be94f3fbd5565.png)

I use top and pidstat to confirm that the process is elasticsearch, and "perf top" shows like this:

```auto
71.51% [kernel] [k] __pv_queued_spin_lock_slowpath
       1.75% [kernel] [k] _raw_spin_lock_irqsave
       1.42% [kernel] [k] compact_checklock_irqsave.isra.24

```

or like this:

```auto
7.89% [kernel] [k] isolate_freepages_block
   3.96% [kernel] [k] __pv_queued_spin_lock_slowpath
   3.63% [kernel] [k] copy_user_enhanced_fast_string
   1.75% [kernel] [k] __list_del_entry

```

Is this a bug, or something else?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 22, 2021, 1:54am UTC](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532/2 "2021-03-22T01:54:51Z")

</div>

What do your hot threads or slow logs or Elasticsearch logs show at this time?

---

<div class="post-metadata">

**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\
**Post date:** [March 22, 2021, 2:44am UTC](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532/3 "2021-03-22T02:44:38Z")

</div>

I used to change the hostname of each node and reinstall ES from version 5.6.3 to version 7.11.1, and then add them to another cluster.  
After rebooting the system 2 days ago, everything is ok now.I forgot to get the hot thread info.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2021, 2:45am UTC](https://discuss.elastic.co/t/100-cpu-system-time-used-on-hdd-data-node/267532/4 "2021-04-19T02:45:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.