# Only one node in cluster(6 nodes) slow query and high read I/O when querying

**URL:** <https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158>\
**Category:** Elasticsearch\
**Created:** [July 30, 2020, 5:13am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158 "2020-07-30T05:13:54Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sunghoon\_Jo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunghoon_jo/32/46276_2.png) [@Sunghoon\_Jo](https://discuss.elastic.co/u/Sunghoon_Jo)\
**Post date:** [July 30, 2020, 5:13am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/1 "2020-07-30T05:13:54Z")

</div>

Hi All,

I have some problem when querying. My cluster have 6 data nodes.  
When I query, only one node has unusual symptoms. my query is very simple.  
I'm doing stress tests, only one node shows high Read I/O(3000/s) , Latency  
And Unlike other nodes, It shows a low CPU usage.

```auto
Issue node Max: 
  - CPU: 12%
  - Read I/O: 3000/s
  - Latency: 500ms
  - Cgroup CPU Performance: 600m ns

Others Max
  - CPU: 33%
  - Read I/O: 10/s
  - Latency: 10ms
  - Cgroup CPU Performance: 2b ns

```

Why is this happening?

This is my query

```auto
{
    "sort": [
        {
            "rank_score": "desc"
        }
    ],
    "from": 0,
    "size": 1005,
    "_source": false,
    "docvalue_fields": [
        "attributes","acronym","channel_id",
        "channel_name","channel_number","lineup_id",
        "original_logo_image_url","source_id","recency_norm",
        "popularity_norm","search_popularity_norm"
    ],
    "query": {
        "constant_score": {
            "filter": {
                "bool": {
                    "must": [
                        {
                            "term": {
                                "lineup_id": {
                                    "value": "42258"
                                }
                            }
                        }
                    ]
                }
            },
            "boost": 1
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 30, 2020, 5:20am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/2 "2020-07-30T05:20:37Z")

</div>

Are you distributing queries evenly across the cluster? Are queried shards and indices evenly distributed across the cluster? Which version of Elasticsearch are you using?

---

<div class="post-metadata">

**Author:** ![Sunghoon\_Jo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunghoon_jo/32/46276_2.png) [@Sunghoon\_Jo](https://discuss.elastic.co/u/Sunghoon_Jo)\
**Post date:** [July 30, 2020, 5:21am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/3 "2020-07-30T05:21:48Z")

</div>

Hello Christian\_Dahlqvist

Primary shard: 1  
Replica shard: 5

One node have one shard. and my elasticsearch version is 7.6.1

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 30, 2020, 6:48am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/4 "2020-07-30T06:48:36Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Are you distributing queries evenly across the cluster?

How are you running your test?

If distribution is even it may be worthwhile checking for hardware problems on the node in question.

---

<div class="post-metadata">

**Author:** ![Sunghoon\_Jo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunghoon_jo/32/46276_2.png) [@Sunghoon\_Jo](https://discuss.elastic.co/u/Sunghoon_Jo)\
**Post date:** [July 30, 2020, 8:01am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/5 "2020-07-30T08:01:30Z")

</div>

Query for random numbers.  
Also, the instance is using AWS. Even if you delete a specific instance, it shows the same symptoms in other nodes

this is what happens when I have 2 shards.

```auto
Issue nodes(2 nodes) Max: 
  - CPU: 15%
  - Read I/O: 3000/s
  - Latency: 500ms

Others Max
  - CPU: 95%
  - Read I/O: 2500/s
  - Latency: 700ms

```

I think it's related to the ElasticSearch mechanism.  
Does ES drive tasks such as Disk I / O to a particular node?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 30, 2020, 8:09am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/6 "2020-07-30T08:09:30Z")

</div>

I can not tell based on the information you have provided so far. I think we need more details about how the cluster is set up and how you are running the test.

---

<div class="post-metadata">

**Author:** ![Sunghoon\_Jo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunghoon_jo/32/46276_2.png) [@Sunghoon\_Jo](https://discuss.elastic.co/u/Sunghoon_Jo)\
**Post date:** [July 30, 2020, 8:17am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/7 "2020-07-30T08:17:23Z")

</div>

Cluster

```auto
AWS EKS (c5.2xlarge * 6 instance)
master: 3 nodes
data: 6 nodes
indices: 1 Primary, 5 Replicas.
Request per sec: 60

```

In the query below, only "value" are changed to random.

```auto
{
    "sort": [{ "rank_score": "desc" }],
    "from": 0,
    "size": 1005,
    "_source": false,
    "docvalue_fields": [
        "attributes","acronym","channel_id",
        "channel_name","channel_number","lineup_id",
        "original_logo_image_url","source_id","recency_norm",
        "popularity_norm","search_popularity_norm"
    ],
    "query": {
        "constant_score": {
            "filter": {
                "bool": {
                    "must": [
                        {
                            "term": {
                                "lineup_id": {
                                    "value": "42258" <- changed this line
                                }
                            }
                        }
                    ]
                }
            },
            "boost": 1
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2020, 8:17am UTC](https://discuss.elastic.co/t/only-one-node-in-cluster-6-nodes-slow-query-and-high-read-i-o-when-querying/243158/8 "2020-08-27T08:17:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.