# Master node with 100% CPU

**URL:** <https://discuss.elastic.co/t/master-node-with-100-cpu/263003>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [February 2, 2021, 2:58pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003 "2021-02-02T14:58:02Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 2, 2021, 2:58pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/1 "2021-02-02T14:58:02Z")

</div>

we are getting 100% CPU usage for a master node.  
the cluster is hosted in the Elastic cloud.

any help will be appreciated!

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8beff32a03dddf4297734a1b1c160cd13d917c96.png)  
attached below the tasks (it seems to run /admin/snapshot/delete constantly )

![image](https://us1.discourse-cdn.com/elastic/original/3X/e/d/edbde6e5ef6baadd8dd9fd7692a5098910458758.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/8/281a3e809b54a81eef2c3a4062c5b654182fd565.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/1/41b1733a91a88d2523df53854635a8cbcb7d2d80.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/5/9598e443ecd870e38da5422e75c3be3781952e3a.png)

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 2, 2021, 4:30pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/2 "2021-02-02T16:30:07Z")

</div>

not sure about this snapshot task (not sure if it's actually a snapshot- same one you'll find in Kibana.. or maybe it's talking about memory snapshot or similar), but after changing the snapshot policy in Kibana , the retention policy, to delete after 10000 snapshots / 24 days, instead of 100 and 3 days, the load on the master node stopped.

it will be great to understand this issue, as it may be a bug in the cloud / Elastic etc

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 2, 2021, 6:57pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/3 "2021-02-02T18:57:56Z")

</div>

and the load is back.. not sure why I always see something in the running tasks related to the snapshot even that the interval is 1 hour (for example /admin/snapshot/status- 45min duration!)

 ![hot_threads2](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b04bb2ebad1fd7f69773219c2d3f5dfdcad4fd45.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 3, 2021, 3:54am UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/4 "2021-02-03T03:54:02Z")

</div>

What is the output from the `_cluster/stats?pretty&human` API?

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 3, 2021, 8:44am UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/5 "2021-02-03T08:44:57Z")

</div>

attached. thanks a lot @warkolm

> <https://gist.github.com/lior-fortifieddata/689e66fe5909d5a9a08f751a7731c924>

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 3, 2021, 11:04am UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/6 "2021-02-03T11:04:32Z")

</div>

I'm still not sure why I see in the node's tasks the following actions all the time:

```
"action" : "cluster:admin/snapshot/status"
 "action" : "cluster:admin/snapshot/get",
 "action" : "cluster:admin/snapshot/delete",

```

while the (only) snapshot policy says

**Schedule**  
**0 0 \* \* \* ?**

```
 {
    "Mdnq4RAeRtqs004FxxQRgg:1340468" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340468,
      "type" : "transport",
      "action" : "cluster:monitor/xpack/ml/job/stats/get",
      "description" : "",
      "start_time_in_millis" : 1612349554476,
      "running_time_in_nanos" : 1308251137,
      "cancellable" : false,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:466567" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 466567,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/status",
      "description" : "",
      "start_time_in_millis" : 1612316979079,
      "running_time_in_nanos" : 32576705363709,
      "cancellable" : false,
      "parent_task_id" : "Ysr-3BHkT9aKvQXSQdtr2A:1225810",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340470" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340470,
      "type" : "transport",
      "action" : "indices:monitor/stats",
      "description" : "",
      "start_time_in_millis" : 1612349554477,
      "running_time_in_nanos" : 1307514234,
      "cancellable" : false,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340496" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340496,
      "type" : "transport",
      "action" : "cluster:monitor/tasks/lists[n]",
      "description" : "",
      "start_time_in_millis" : 1612349555784,
      "running_time_in_nanos" : 321901,
      "cancellable" : false,
      "parent_task_id" : "xLr4tF2ZSSKNscRDwYLllA:1954171",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:357473" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 357473,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/status",
      "description" : "",
      "start_time_in_millis" : 1612313393273,
      "running_time_in_nanos" : 36162511051383,
      "cancellable" : false,
      "parent_task_id" : "Ysr-3BHkT9aKvQXSQdtr2A:959206",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:962091" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 962091,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/status",
      "description" : "",
      "start_time_in_millis" : 1612334913474,
      "running_time_in_nanos" : 14642310541533,
      "cancellable" : false,
      "parent_task_id" : "Ysr-3BHkT9aKvQXSQdtr2A:2495902",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340476" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340476,
      "type" : "transport",
      "action" : "indices:data/read/search",
      "description" : """indices[.security], types[], search_type[QUERY_THEN_FETCH], scroll[5m], source[{"size":1000,"query":{"term":{"doc_type":{"value":"role-mapping","boost":1.0}}},"_source":{"includes":[],"excludes":[]}}]""",
      "start_time_in_millis" : 1612349554873,
      "running_time_in_nanos" : 910793110,
      "cancellable" : true,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340477" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340477,
      "type" : "transport",
      "action" : "indices:data/read/search",
      "description" : """indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"query":{"term":{"type":{"value":"user","boost":1.0}}},"track_total_hits":2147483647}]""",
      "start_time_in_millis" : 1612349554874,
      "running_time_in_nanos" : 910431309,
      "cancellable" : true,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1328664" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1328664,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/delete",
      "description" : "",
      "start_time_in_millis" : 1612349116182,
      "running_time_in_nanos" : 439601908076,
      "cancellable" : false,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340472" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340472,
      "type" : "transport",
      "action" : "indices:data/read/msearch",
      "description" : """,indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"query":{"term":{"type":{"value":"role","boost":1.0}}},"track_total_hits":2147483647}]indices:data/read/msearch[indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"terminate_after":1,"query":{"bool":{"must":[{"term":{"type":{"value":"role","boost":1.0}}},{"bool":{"should":[{"exists":{"field":"indices.field_security.grant","boost":1.0}},{"exists":{"field":"indices.field_security.except","boost":1.0}},{"exists":{"field":"indices.fields","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"track_total_hits":2147483647}]indices:data/read/msearch[indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"terminate_after":1,"query":{"bool":{"must":[{"term":{"type":{"value":"role","boost":1.0}}}],"filter":[{"exists":{"field":"indices.query","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"track_total_hits":2147483647}]]""",
      "start_time_in_millis" : 1612349554780,
      "running_time_in_nanos" : 1005474302,
      "cancellable" : true,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340473" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340473,
      "type" : "transport",
      "action" : "indices:data/read/search",
      "description" : """indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"query":{"term":{"type":{"value":"role","boost":1.0}}},"track_total_hits":2147483647}]""",
      "start_time_in_millis" : 1612349554780,
      "running_time_in_nanos" : 1005439702,
      "cancellable" : true,
      "parent_task_id" : "Mdnq4RAeRtqs004FxxQRgg:1340472",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340474" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340474,
      "type" : "transport",
      "action" : "indices:data/read/search",
      "description" : """indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"terminate_after":1,"query":{"bool":{"must":[{"term":{"type":{"value":"role","boost":1.0}}},{"bool":{"should":[{"exists":{"field":"indices.field_security.grant","boost":1.0}},{"exists":{"field":"indices.field_security.except","boost":1.0}},{"exists":{"field":"indices.fields","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"track_total_hits":2147483647}]""",
      "start_time_in_millis" : 1612349554780,
      "running_time_in_nanos" : 1004923500,
      "cancellable" : true,
      "parent_task_id" : "Mdnq4RAeRtqs004FxxQRgg:1340472",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340475" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340475,
      "type" : "transport",
      "action" : "indices:data/read/search",
      "description" : """indices[.security], types[], search_type[QUERY_THEN_FETCH], source[{"size":0,"terminate_after":1,"query":{"bool":{"must":[{"term":{"type":{"value":"role","boost":1.0}}}],"filter":[{"exists":{"field":"indices.query","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"track_total_hits":2147483647}]""",
      "start_time_in_millis" : 1612349554781,
      "running_time_in_nanos" : 1004689399,
      "cancellable" : true,
      "parent_task_id" : "Mdnq4RAeRtqs004FxxQRgg:1340472",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:873372" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 873372,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/status",
      "description" : "",
      "start_time_in_millis" : 1612331322773,
      "running_time_in_nanos" : 18233011841285,
      "cancellable" : false,
      "parent_task_id" : "Ysr-3BHkT9aKvQXSQdtr2A:2250652",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340455" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340455,
      "type" : "direct",
      "action" : "internal:cluster/coordination/publish_state",
      "description" : "",
      "start_time_in_millis" : 1612349554277,
      "running_time_in_nanos" : 1508437753,
      "cancellable" : false,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340419" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340419,
      "type" : "transport",
      "action" : "indices:admin/template/put",
      "description" : "",
      "start_time_in_millis" : 1612349553374,
      "running_time_in_nanos" : 2411770240,
      "cancellable" : false,
      "parent_task_id" : "n86rCLPyT-ebf6e4uaMNew:2893588",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:777628" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 777628,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/status",
      "description" : "",
      "start_time_in_millis" : 1612327726073,
      "running_time_in_nanos" : 21829712341262,
      "cancellable" : false,
      "parent_task_id" : "Ysr-3BHkT9aKvQXSQdtr2A:1984186",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1330988" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1330988,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/status",
      "description" : "",
      "start_time_in_millis" : 1612349202279,
      "running_time_in_nanos" : 353506669293,
      "cancellable" : false,
      "parent_task_id" : "Ysr-3BHkT9aKvQXSQdtr2A:3515287",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1339885" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1339885,
      "type" : "transport",
      "action" : "cluster:monitor/xpack/usage",
      "description" : "",
      "start_time_in_millis" : 1612349532976,
      "running_time_in_nanos" : 22809192767,
      "cancellable" : false,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340463" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340463,
      "type" : "transport",
      "action" : "indices:admin/template/put",
      "description" : "",
      "start_time_in_millis" : 1612349554374,
      "running_time_in_nanos" : 1411500354,
      "cancellable" : false,
      "parent_task_id" : "xLr4tF2ZSSKNscRDwYLllA:1954036",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1339688" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1339688,
      "type" : "transport",
      "action" : "cluster:admin/snapshot/get",
      "description" : "",
      "start_time_in_millis" : 1612349527879,
      "running_time_in_nanos" : 27906168691,
      "cancellable" : false,
      "parent_task_id" : "aIsgB69pTfCzDRhxS2qqEw:2724639",
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340490" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340490,
      "type" : "transport",
      "action" : "cluster:monitor/xpack/usage",
      "description" : "",
      "start_time_in_millis" : 1612349555776,
      "running_time_in_nanos" : 9515129,
      "cancellable" : false,
      "headers" : { }
    },
    "Mdnq4RAeRtqs004FxxQRgg:1340491" : {
      "node" : "Mdnq4RAeRtqs004FxxQRgg",
      "id" : 1340491,
      "type" : "transport",
      "action" : "cluster:monitor/nodes/stats",
      "description" : "",
      "start_time_in_millis" : 1612349555776,
      "running_time_in_nanos" : 9364729,
      "cancellable" : false,
      "headers" : { }
    }
  }
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 3, 2021, 9:06pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/7 "2021-02-03T21:06:55Z")

</div>

Thanks! Can you post the hot threads from the master as well please?

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 3, 2021, 9:30pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/8 "2021-02-03T21:30:44Z")

</div>

the above is from the master node. this is the node I was referring to, that has 100% CPU. the others are kind of sleeping..

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/5/65da166bdf99d3a2d0587773eadabac28c5312e6.png)

current state:

> <https://gist.github.com/lior-fortifieddata/bce11b8eef2fa8a3f1f9c638efbe10fd>

tasks:

> <https://gist.github.com/lior-fortifieddata/91adfa97c562c5c676441c49838174ed>

Thanks for looking

\*\* I'm trying now to upgrade the masters, and also to freeze old indices

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 4, 2021, 4:19am UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/9 "2021-02-04T04:19:57Z")

</div>

There's not a tonne jumping out to me. Have you tried raising an issue with the Support team?

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [February 10, 2021, 9:36pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/10 "2021-02-10T21:36:32Z")

</div>

It was due to an increase in index rate. An upgrade of the master nodes resolved it  
Thanks a lot

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 10, 2021, 9:36pm UTC](https://discuss.elastic.co/t/master-node-with-100-cpu/263003/11 "2021-03-10T21:36:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.