# Cluster suddenly blew up out of nowhere

**URL:** <https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017>\
**Category:** Elasticsearch\
**Created:** [December 24, 2018, 5:10pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017 "2018-12-24T17:10:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![baller](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@baller](https://discuss.elastic.co/u/baller)\
**Post date:** [December 24, 2018, 5:10pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/1 "2018-12-24T17:10:36Z")

</div>

Hi all,

Approximately 1 hour ago we started getting connection and indexing errors on our cluster 36b401. We didn't make any config changes or any other changes to our app. After a few minutes of reattempting connections from various places, we issued a force restart command through the elastic website. That restart has been running for over an hour now and only one of our two instances has come back.

Is anyone experiencing issues like this? I don't see Elastic reporting any issues and I've also emailed support. Thanks.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 24, 2018, 5:16pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/2 "2018-12-24T17:16:03Z")

</div>

What is the output of the cluster health API?

---

<div class="post-metadata">

**Author:** ![baller](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@baller](https://discuss.elastic.co/u/baller)\
**Post date:** [December 24, 2018, 5:18pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/3 "2018-12-24T17:18:59Z")

</div>

Output below. Another note: we haven't seen any improvement on the unassigned shards number for at least 20-30 minutes. Might need to do an even harder restart of the cluster?

```
{
  "active_primary_shards": 107,
  "active_shards": 107,
  "active_shards_percent_as_number": 49.76744186046512,
  "cluster_name": "36b401<redacted>",
  "delayed_unassigned_shards": 0,
  "initializing_shards": 0,
  "number_of_data_nodes": 1,
  "number_of_in_flight_fetch": 0,
  "number_of_nodes": 2,
  "number_of_pending_tasks": 0,
  "relocating_shards": 0,
  "status": "red",
  "task_max_waiting_in_queue_millis": 0,
  "timed_out": false,
  "unassigned_shards": 108
}
```

---

<div class="post-metadata">

**Author:** ![baller](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@baller](https://discuss.elastic.co/u/baller)\
**Post date:** [December 24, 2018, 5:50pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/4 "2018-12-24T17:50:11Z")

</div>

Is there anything else we can do to move along this stuck restart?

---

<div class="post-metadata">

**Author:** ![baller](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@baller](https://discuss.elastic.co/u/baller)\
**Post date:** [December 24, 2018, 6:13pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/5 "2018-12-24T18:13:06Z")

</div>

I cancelled the restart to see if that helps launch the failed node.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 24, 2018, 6:16pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/6 "2018-12-24T18:16:43Z")

</div>

What is the size of your cluster? Do you have monitoring enabled?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 21, 2019, 6:30pm UTC](https://discuss.elastic.co/t/cluster-suddenly-blew-up-out-of-nowhere/162017/7 "2019-01-21T18:30:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.