# Data node CPU utilization goes very high

**URL:** <https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464>\
**Category:** Elasticsearch\
**Created:** [August 18, 2020, 4:02pm UTC](https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464 "2020-08-18T16:02:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sheetal.nainwal](https://avatars.discourse-cdn.com/v4/letter/s/df705f/32.png) [@sheetal.nainwal](https://discuss.elastic.co/u/sheetal.nainwal)\
**Post date:** [August 18, 2020, 4:02pm UTC](https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464/1 "2020-08-18T16:02:25Z")

</div>

I am using AWS ElasticSearch service and the problem I am facing while ingestion happens on one index, it serves Very high CPU utilization (sometimes 1 or 2 master/data node goes down and sometimes ingestion fails). As result it logs warnings/errors as:

`[WARN][o.e.c.NodeConnectionsService] [7d6f3c47d7f582ced2c090fbf6a3afe5] failed to connect to node {a80c4027a8ff7917bd4f7h8j9k8g5f4d}{PsiWPcLZQEarONo143BLzQ}{ER6LljwCQfGnD4nm42n2TQ}{ __IP__ }{ __IP__ }{distributed_snapshot_deletion_enabled=true, __AMAZON_INTERNAL__ , __AMAZON_INTERNAL__ , cross_cluster_transport_address= __IP__ } (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [a80c4027a8ff7917bc0c3767dde0f72e][__IP__] connect_exception at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.1.jar:7.1.1] indent preformatted text by 4 spaces`

`[WARN][o.e.t.TransportService] [0b1fd2f1876cdcee4abd7a1dcee545454f] Received response for a request that has timed out, sent [18406ms] ago, timed out [3401ms] ago, action [__PATH__ [n]], node [{784e7ea9800931208c1a36c04db940e3}{SITLwWDBTpKr2kKrHvkIRQ}{Gq22BKc7SYSuorfCb5lrKA}{ __IP__ }{ __IP__ }{distributed_snapshot_deletion_enabled=true, __AMAZON_INTERNAL__ , __AMAZON_INTERNAL__ , cross_cluster_transport_address= __IP__ }], id [3297456]`

My ES cluster have :  
Data Nodes : 3 (i3.2xlarge.elasticsearch)  
Master Nodes: 3 (c5.xlarge.elasticsearch)  
Number of Indexes : 18 (Average size of each Index is 15 GB)

Where 9 indexes have 1 Primary and 2 Replica shards and rest Indexes have 2 Primary and 2 Replica shards.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 18, 2020, 9:03pm UTC](https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464/2 "2020-08-18T21:03:48Z")

</div>

Welcome to our community! 😃 Please note that AWS ES is a fork of what we offer, so we may not be able to provide support for this problem.

What sort of monitoring functionality do you have in place?

---

<div class="post-metadata">

**Author:** ![sheetal.nainwal](https://avatars.discourse-cdn.com/v4/letter/s/df705f/32.png) [@sheetal.nainwal](https://discuss.elastic.co/u/sheetal.nainwal)\
**Post date:** [August 19, 2020, 6:05am UTC](https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464/3 "2020-08-19T06:05:25Z")

</div>

Thank you Mark and yes you are right but I wanted a bit more insights on what could have happened to data node when it starts consuming more CPU

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 19, 2020, 6:06am UTC](https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464/4 "2020-08-19T06:06:42Z")

</div>

Ok, so;

> [@warkolm](#):
>
> What sort of monitoring functionality do you have in place?

Have you looked at hot threads?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 16, 2020, 6:06am UTC](https://discuss.elastic.co/t/data-node-cpu-utilization-goes-very-high/245464/5 "2020-09-16T06:06:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.