# ES 2.4 failed to discover master after hard disks ran full

**URL:** <https://discuss.elastic.co/t/es-2-4-failed-to-discover-master-after-hard-disks-ran-full/152838>\
**Category:** Elasticsearch\
**Created:** [October 17, 2018, 12:39pm UTC](https://discuss.elastic.co/t/es-2-4-failed-to-discover-master-after-hard-disks-ran-full/152838 "2018-10-17T12:39:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nilsnolde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nilsnolde/32/36611_2.png) [@nilsnolde](https://discuss.elastic.co/u/nilsnolde)\
**Post date:** [October 17, 2018, 12:39pm UTC](https://discuss.elastic.co/t/es-2-4-failed-to-discover-master-after-hard-disks-ran-full/152838/1 "2018-10-17T12:39:53Z")

</div>

Hi,

I'm fairly new to ES and have to use 2.4 within the framework of [Pelias](https://github.com/pelias/pelias).

A few days ago, I realized that ES wrote logs totalling \> 900GB on the entire cluster, both to `log` directory and syslog. Resulting in all disks (root and ES data mounted one) running completely full (I installed netdata now, so I'll get warned next time). That was mostly due to Pelias (well, and ES VERY verbose error logs). Anyways, I knew smth would be messed up after I cleaned the logs directories.

Now, my cluster is responding to `/_cluster/health?pretty` queries (or `/_cat/indices/v`) with

```json
{
  "error" : {
    "root_cause" : [ {
      "type" : "master_not_discovered_exception",
      "reason" : null
    } ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

```

What does work is `/_stats?pretty` (output is too long for a post). But that's not super useful for my particular problem. However, I did notice that the main index (distributed over 24 shards) has only a fraction of the documents it should have (23 Mio instead of \> 500 Mio). Which looks to me like a lot of shards went offline.. But then again, can't query any `/_cat` or `/<index>/settings` API..

The API which is querying ES in the background, i.e. the Pelias API, randomly succeeds or throws this:

`"[cluster_block_exception] blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"`

As I said, I'm not very used to ES and this definitely exceeds my knowledge. If possible, I want to recover the cluster without destroying it entirely (if not too late already) and set it up from scratch.

Can you pls give me hints how to proceed or debug?

Setup:

- Ubuntu 16.04
- ES v2.4
- 4 node cluster with 32GB RAM each and 16GB heap size for ES

Many thanks  
Nils

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2018, 12:39pm UTC](https://discuss.elastic.co/t/es-2-4-failed-to-discover-master-after-hard-disks-ran-full/152838/2 "2018-11-14T12:39:55Z")

</div>

Closed on November 14, 2018, 12:39pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.