# Cluster down after an autoreboot?

**URL:** <https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890>\
**Category:** Elasticsearch\
**Created:** [February 7, 2018, 5:14pm UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890 "2018-02-07T17:14:20Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nicholas\_Thompson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicholas_thompson/32/18700_2.png) [@Nicholas\_Thompson](https://discuss.elastic.co/u/Nicholas_Thompson)\
**Post date:** [February 7, 2018, 5:14pm UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890/1 "2018-02-07T17:14:21Z")

</div>

Hi,

We got an email at 1418 saying the cluster had been automatically rebooted due to high memory usage.

We can now see 2 instances in the cluster (there was 1 before) and one of them (001) just keeps rebooting itself. The other (002) seems to be complaining in the logs about being unable to find a master node?

We've raised a support ticket but the SLA says up to 3 business days.

Anyone got any suggestions on what we can do in the meantime?

Nick

---

<div class="post-metadata">

**Author:** ![Nicholas\_Thompson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicholas_thompson/32/18700_2.png) [@Nicholas\_Thompson](https://discuss.elastic.co/u/Nicholas_Thompson)\
**Post date:** [February 7, 2018, 5:33pm UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890/2 "2018-02-07T17:33:26Z")

</div>

Seems to have magically come back up on its own? Unless the support team have claimed the ticket?

We still have a mystery second instance...

![image](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2dee2f23f52b0dae9611f73ca9d3c74f12830bf2.png)

Is 278 unassigned shards bad?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 7, 2018, 6:05pm UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890/3 "2018-02-07T18:05:43Z")

</div>

How large are the instances? How many shards do you have in total?

---

<div class="post-metadata">

**Author:** ![Nicholas\_Thompson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicholas_thompson/32/18700_2.png) [@Nicholas\_Thompson](https://discuss.elastic.co/u/Nicholas_Thompson)\
**Post date:** [February 7, 2018, 10:35pm UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890/4 "2018-02-07T22:35:31Z")

</div>

It's running a single node, 2Gb RAM instance. According to `_cat/health`, there are 279 shards and 278 unassigned?!

In terms of data, I thought we had about 4 indexes (total of less than 1.5m rows and ~1Gb data)... however `_cat/indices` shows another 260 indices for `.watcher-history...` dating back to May 2017 (and a few small kibana tables). Would these cause any issues?!

Checking the cluster again this evening and we're back down to a single instance running at 60% mem pressure (although the graph indicates it frequently hits the 70% limit and does some kind of GC?)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 8, 2018, 6:21am UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890/5 "2018-02-08T06:21:32Z")

</div>

If you have replicas configured for any of the indices, Elasticsearch will not be able to place these as you only have 1 node. That is nothing to worry about.

Having all those `.watcher-history...` indices around will use up resources, so I would recommend deleting older ones, e.g. all the ones from last year or the ones older than a month ago. This will help with reducing heap pressure.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2018, 6:21am UTC](https://discuss.elastic.co/t/cluster-down-after-an-autoreboot/118890/6 "2018-03-08T06:21:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
