# Flush threads stuck \[ES 6.1.2\]

**URL:** <https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707>\
**Category:** Elasticsearch\
**Created:** [January 30, 2018, 9:44pm UTC](https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707 "2018-01-30T21:44:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\
**Post date:** [January 30, 2018, 9:44pm UTC](https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707/1 "2018-01-30T21:44:31Z")

</div>

I have run into an issue a few times now where a node is not receiving any requests but is using ~50% CPU indefinitely.  
Looking at `GET /_cat/thread_pool?v`, I see 1 or more active flush thread. This node seems to have 4 stuck in flush:

```auto
node_name name active queue rejected
node-b flush 4 0 0

```

The output of `hot_threads` corroborates this:

```auto
   98.0% (489.9ms out of 500ms) cpu usage by thread 'elasticsearch[node-b][flush][T#2]'
   97.8% (489.1ms out of 500ms) cpu usage by thread 'elasticsearch[node-b][flush][T#3]'
   97.7% (488.5ms out of 500ms) cpu usage by thread 'elasticsearch[node-b][flush][T#4]'

```

In these cases, running `POST /_flush/synced?pretty` gives an error on one or more shards like this:

```auto
"my_index": {
    "total": 6,
    "successful": 4,
    "failed": 2,
    "failures": [
      {
        "shard": 1,
        "reason": "[1] ongoing operations on primary"
      }
    ]
  }

```

Restarting the node fixes the problem, but I'd like to understand how it gets into this state, and how I might keep this from happening in the first place.

There is certainly a lot of shard migration happening on this cluster, so perhaps that has something to do with it.

Is there anything else I should look at to diagnose this, or anything else I can do to rectify the situation other than bouncing ES?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 31, 2018, 8:52am UTC](https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707/2 "2018-01-31T08:52:18Z")

</div>

Hey,

not a hundred percent sure yet, you might be hitting this one (fixed in 6.1.3): [https://github.com/elastic/elasticsearch/pull/28350](https://github.com/elastic/elasticsearch/pull/28350)

--Alex

---

<div class="post-metadata">

**Author:** ![nhat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhat/32/22170_2.png) [@nhat](https://discuss.elastic.co/u/nhat)\
**Post date:** [January 31, 2018, 2:06pm UTC](https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707/3 "2018-01-31T14:06:19Z")

</div>

@loren Could you please provide us a shard-level stats? This can be retrieved via `GET /_stats?level=shards`. You can post it here or send it to me at [nhat.nguyen@elastic.co](mailto:nhat.nguyen@elastic.co). Thank you!

---

<div class="post-metadata">

**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\
**Post date:** [January 31, 2018, 7:46pm UTC](https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707/4 "2018-01-31T19:46:25Z")

</div>

That looks awfully similar to the behavior we're seeing. Upgrading right now. I'll report back with shard-level stats if I see it again. Thank you for the heads up!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2018, 7:46pm UTC](https://discuss.elastic.co/t/flush-threads-stuck-es-6-1-2/117707/5 "2018-02-28T19:46:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.