# Shard stuck in STORE TRANSLOG stage

**URL:** <https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112>\
**Category:** Elasticsearch\
**Created:** [December 29, 2015, 2:58pm UTC](https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112 "2015-12-29T14:58:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Omar\_Al\_Zabir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omar_al_zabir/32/5943_2.png) [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\
**Post date:** [December 29, 2015, 2:58pm UTC](https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112/1 "2015-12-29T14:58:53Z")

</div>

My topbeat index has two shards 0, 1 which are both stuck at TRANSLOG stage for hours. CLuster state is red. Please help me troubleshoot this problem:

```
"id": 1,
  "type": "STORE",
  "stage": "TRANSLOG",
  "primary": true,
  "start_time": "2015-12-29T13:03:04.160Z",
  "start_time_in_millis": 1451394184160,
  "total_time": "1.9h",
  "total_time_in_millis": 6847526,
  "source": {
    "id": "RWxvylvLQcSF4T3RM6xG8A",
    "host": "10.35.132.142",
    "transport_address": "10.35.132.142:9300",
    "ip": "10.35.132.142",
    "name": "ec-dyl09026app03"
  },
  "target": {
    "id": "RWxvylvLQcSF4T3RM6xG8A",
    "host": "10.35.132.142",
    "transport_address": "10.35.132.142:9300",
    "ip": "10.35.132.142",
    "name": "ec-dyl09026app03"
  },
  "index": {
    "size": {
      "total": "296.9mb",
      "total_in_bytes": 311398655,
      "reused": "296.9mb",
      "reused_in_bytes": 311398655,
      "recovered": "0b",
      "recovered_in_bytes": 0,
      "percent": "100.0%"
    },
    "files": {
      "total": 88,
      "reused": 88,
      "recovered": 0,
      "percent": "100.0%"
    },
    "total_time": "26ms",
    "total_time_in_millis": 26,
    "source_throttle_time": "-1",
    "source_throttle_time_in_millis": 0,
    "target_throttle_time": "-1",
    "target_throttle_time_in_millis": 0
  },
  "translog": {
    "recovered": 379391,
    "total": -1,
    "percent": "-1.0%",
    "total_on_start": -1,
    "total_time": "1.9h",
    "total_time_in_millis": 6847500
  },
  "verify_index": {
    "check_index_time": "0s",
    "check_index_time_in_millis": 0,
    "total_time": "0s",
    "total_time_in_millis": 0
  }
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 29, 2015, 11:16pm UTC](https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112/2 "2015-12-29T23:16:31Z")

</div>

Check your ES logs.

What version are you on?

---

<div class="post-metadata">

**Author:** ![Omar\_Al\_Zabir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/omar_al_zabir/32/5943_2.png) [@Omar\_Al\_Zabir](https://discuss.elastic.co/u/Omar_Al_Zabir)\
**Post date:** [December 30, 2015, 10:29am UTC](https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112/3 "2015-12-30T10:29:24Z")

</div>

ES 2.1.0.

I tried many things:

- Tried closing the index and reopening, no luck.
- Tried deleting index and recreating, no luck. As soon as new shards are created, they get stuck.
- Various shards started going into unassigned mode and never getting assigned, despite many hours.
- I noticed the EC process consumed 100% CPU continuously for hours. So, they must have gone into some infinite loop.
- New indexes started getting into STORE mode.
- Tried rolling restart of EC nodes, did not help.
- Tried stopping all EC nodes and starting them all in one shot. Did not help either.

All these started happening, when I tried to close a couple of old indices.

After many hours of struggle, I finally deleted the data folder. Now everything is running fine.

If you are interested, I can give you the logs. It should reveal issues with bulk closing of index causing the cluster to get into irrecoverable state.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 1, 2016, 3:52am UTC](https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112/4 "2016-01-01T03:52:06Z")

</div>

If you kept track of everything it may be worth raising an issue on Github.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:27pm UTC](https://discuss.elastic.co/t/shard-stuck-in-store-translog-stage/38112/5 "2017-07-05T23:27:33Z")

</div>