# High CPU Pause Time **URL:** \ **Category:** Elasticsearch\ **Created:** [June 27, 2016, 1:45pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046 "2016-06-27T13:45:58Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![german23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/german23/32/11052_2.png) [@german23](https://discuss.elastic.co/u/german23)\ **Post date:** [June 27, 2016, 1:45pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/1 "2016-06-27T13:45:58Z")
Hi guys, we got a quite strange problem on one of our elasticsearch server - every 20-40sec ES stops using CPU resources and get in a "pause" time - see screenshot below. ![](https://us1.discourse-cdn.com/elastic/original/2X/c/c54d1aa7f7d58f48bb079a549a61fdbf9dd3b338.PNG) About the Environment: -64CPU Cores, 256GB RAM, ES 1.7.5 (Upgrading is NOT a choice atm), 4 Instances a 30GB Heap Running on the server, Rest is Cache for the ZFS-Filesystem that is installed on the SSDs where the data is. My first thought was that there are big GCs running and the GC pauses the ES-Application, however i turned on debug logging and set the GC log times quite low and there is no crazy 15sec Garbagecollection running when the CPU Pause times appear. Since we got 4 hardware identical server with completely identical config and i could not expain why this happen on only 1 specific server, so i went on and reinstalled the Debian Jessie on this machine to make sure it is no OS-Problem - without any success. I also checked if there is any system process spawning when the phenomenon is happening, but again there is nothing running except the 4 ES Instances. Anyone got a tipp for me to where start debugging this? Thanks a lot ---
**Author:** ![german23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/german23/32/11052_2.png) [@german23](https://discuss.elastic.co/u/german23)\ **Post date:** [June 27, 2016, 2:09pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/2 "2016-06-27T14:09:51Z")
I also turned on the TRACE Logging now. I am by far not a "specialist" in reading those complex logs, but it seems as soon as the "Pause Times" starts, an Lucene Merge Thread get spawned. [logstash-2016.06.27][7] elasticsearch[nodename][[logstash-2016.06.27][7]: Lucene Merge Thread #77] TMP: maybe=\_3ueo(4.10.4):c66003 \_4bxm(4.10.4):c54538 \_4cae(4.10.4):c50152 \_4cas(4.10.4):c47271 \_4ch2(4.10.4):c42140 \_4cgl(4.10.4):c36120 \_4cbi(4.10.4):c34035 \_4c7g(4.10.4):c28552 \_4ccg(4.10.4):c27083 \_4cep(4.10.4):c24812 score=0.42923177416322084 skew=0.156 nonDelRatio=1.000 tooLarge=false size=614.593 MB Is it possible that the Merging Process triggers CPU Pause Times maybe? ---
**Author:** ![abeyad](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@abeyad](https://discuss.elastic.co/u/abeyad)\ **Post date:** [June 27, 2016, 8:10pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/3 "2016-06-27T20:10:57Z")
Hello, I would expect merging to take a fair bit of CPU utilization, though it also is heavy on disk I/O. However, I'm not sure it would account for such long CPU pauses. You can find stats for segment merging at `curl -XGET 'http://localhost:9200/_nodes/stats'` (look under the "merge" section). What version of ES are you on? ---
**Author:** ![german23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/german23/32/11052_2.png) [@german23](https://discuss.elastic.co/u/german23)\ **Post date:** [June 28, 2016, 8:44pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/4 "2016-06-28T20:44:53Z")
Hi, thanks for response. I am running ES 1.7.5. If i look into the ongoing merging i see about 10-20Gb of "open" merging and about 15-25 merging threads, that compared to an daily index size of about 1.5-1.9TB seems not much. Anything i should watch in particular that could help me debug this? thanks for response ---
**Author:** ![abeyad](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@abeyad](https://discuss.elastic.co/u/abeyad)\ **Post date:** [June 28, 2016, 9:22pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/5 "2016-06-28T21:22:04Z")
Can you share the I/O wait stats that you're seeing, especially during these pauses? ---
**Author:** ![abeyad](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@abeyad](https://discuss.elastic.co/u/abeyad)\ **Post date:** [June 28, 2016, 9:25pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/6 "2016-06-28T21:25:13Z")
Also, do you have ZFS dedup turned on? If so, turn it off. ---
**Author:** ![german23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/german23/32/11052_2.png) [@german23](https://discuss.elastic.co/u/german23)\ **Post date:** [June 29, 2016, 12:25pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/7 "2016-06-29T12:25:04Z")
We have ZFS Compression with LZ4 activated(small to non-existent overhead) , no dedup though. I see an iowait of 1.2 - 2.8% if i check with iostat linux tool. During the Pauses the iowait "raises" to 3.8-4.9%. I did some more testing and it turns out it is indeed caused by the merging process, it seems like my SSDs cant keep up with the merging if there is heavy index load. I stopped the indexing for about 30 minutes and let the merging process finish ( i saw a huge amount of merging finished debug logs in the es-log) As i now startet the indexing process again, the cpu pauses did not occur again, only if quite some merging accumalate, than the pauses appear again. It seems like i need to add just more nodes to handle all the indexing load or lower the amount of indexing. I really appreciate your help! Regards ---
**Author:** ![abeyad](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@abeyad](https://discuss.elastic.co/u/abeyad)\ **Post date:** [June 29, 2016, 1:31pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/8 "2016-06-29T13:31:41Z")
No problem! Here is a section on the docs with some parameters to try and vary for segment merging and throttling: [https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#segments-and-merging](https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#segments-and-merging) I'm not sure what your indexing rates are, but yes adding more nodes will definitely help with indexing and indeed it may be the case that you've just overloaded your cluster and need to add more servers. I am curious though as to why it is just this one server in your cluster that is experiencing the performance problems. Is there anything different about this server as compared to the others? ---
**Author:** ![german23](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/german23/32/11052_2.png) [@german23](https://discuss.elastic.co/u/german23)\ **Post date:** [June 30, 2016, 1:56pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/9 "2016-06-30T13:56:01Z")
Looking at the hardware of the server, they are 100% identical, however the one that does make probs is the oldest one, maybe SSDs are starting to die or sth like this. I did have a look at your posted link and increased the translog size as well as translog interval and the threshold\_ops - we do not have real time requirements, since it is pure logging case and i got enough heap/memory left, it seems like it is running smoother now a bit. Thank you for your help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:39pm UTC](https://discuss.elastic.co/t/high-cpu-pause-time/54046/10 "2017-07-05T22:39:12Z")