# Elasticsearch cluster corrupted
**URL:** https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721
**Category:** Elasticsearch
**Created:** [December 7, 2017, 5:27pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721 "2017-12-07T17:27:30Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@dualcore](https://discuss.elastic.co/u/dualcore)
#### Post date: [December 7, 2017, 5:27pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/1 "2017-12-07T17:27:30Z")
I deleted my indices on account it filled up my hard drive and I couldn't login to Graylog. Now my Elasticsearch cluster is corrupted. Is there a way to remedy Elasticsearch to restore functionality to Graylog without a reinstall?
I posted the issue with the details and logs on the Graylog forums and was told to ask here for assistance.
> **[Elasticsearch refuses to start](https://community.graylog.org/t/elasticsearch-refuses-to-start/3430/3)**
>
> Elasticsearch \[2017-12-07T00:05:08,206\]\[WARN \]\[o.e.c.r.a.DiskThresholdMonitor\] \[qYyUHAp\] high disk watermark \[90%\] exceeded on \[qYyUHApAREySM8TuBTwSzg\]\[qYyUHAp\]\[/var/lib/elasticsearch/nodes/0\] free: 231.7mb\[0.8%\], shards will be relo$...
Cheers
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 7, 2017, 10:31pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/2 "2017-12-07T22:31:19Z")
You will need to delete everything in the data directory, I don't know where that is as I don't know how graylog installs Elasticsearch. It looks like that would be `/var/lib/elasticsearch`, but you may want to check.
Next time, it's best to use the APIs instead of removing files from the filesystem 🙂
---
### Author:  [@dualcore](https://discuss.elastic.co/u/dualcore)
#### Post date: [December 8, 2017, 8:57pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/3 "2017-12-08T20:57:03Z")
Howdy,
I went ahead an deleted the files in /var/lib/elasticsearch/. Rebooted the machine still in the same situation. I definitely would have deleted them from the API, but the storage was maxed out and refused to start. Catch 22, so to speak. Do you have any other advice to try to get this thing to boot? Also, once it's back up and running from a rebuild or I'm able to save the server, what's the best practice to recycle the logs, so the server doesn't crash again?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 8, 2017, 8:58pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/4 "2017-12-08T20:58:21Z")
What's the actual error?
Does Graylog not provide retention tools? If not, check out [Elasticsearch Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html).
---
### Author:  [@dualcore](https://discuss.elastic.co/u/dualcore)
#### Post date: [December 8, 2017, 9:01pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/5 "2017-12-08T21:01:01Z")
The other errors are on the graylog forums: [Elasticsearch refuses to start - Graylog Central (peer support) - Graylog Community](https://community.graylog.org/t/elasticsearch-refuses-to-start/3430)
The error for checking the status of Elasticsearch:
> Blockquote
> systemctl status elasticsearch.service
> â elasticsearch.service - Elasticsearch
> Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
> Active: failed (Result: exit-code) since Fri 2017-12-08 12:54:23 PST; 24s ago
> Docs: [http://www.elastic.co](http://www.elastic.co)
> Process: 921 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID\_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG\_DIR} -Edefault.path.data=${DATA\_DIR} -Edefault.path.conf=${CONF\_DIR} (code=exited, status=1/FAILURE)
> Process: 914 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
> Main PID: 921 (code=exited, status=1/FAILURE)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.node.Interna...)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.cli.Environm...)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.cli.Environm...)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.cli.Command....)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.cli.Command....)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.bootstrap.El...)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM elasticsearch[921]: at org.elasticsearch.bootstrap.El...)
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM systemd[1]: elasticsearch.service: main process exit...RE
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM systemd[1]: Unit elasticsearch.service entered faile...e.
> Dec 08 12:54:23 SY-DMZ-Graylog-CPEM systemd[1]: elasticsearch.service failed.
> Hint: Some lines were ellipsized, use -l to show in full.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 8, 2017, 9:01pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/6 "2017-12-08T21:01:37Z")
Can you show the logs from Elasticsearch?
---
### Author:  [@dualcore](https://discuss.elastic.co/u/dualcore)
#### Post date: [December 8, 2017, 9:27pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/7 "2017-12-08T21:27:21Z")
This is from the other day when the storage filled up on 12/6/2017, in /var/log/elasticsearch/
graylog-2017-12-06.log
> Blockquote
> GNU nano 2.3.1 File: graylog-2017-12-06.log
> [2017-12-06T10:42:26,712][INFO][o.e.n.Node] initializing ...
> [2017-12-06T10:42:26,883][INFO][o.e.e.NodeEnvironment] [qYyUHAp] using [1] data paths, mounts [[/ (rootfs)]], n$
> [2017-12-06T10:42:26,883][INFO][o.e.e.NodeEnvironment] [qYyUHAp] heap size [1.9gb], compressed ordinary object $
> [2017-12-06T10:42:27,367][INFO][o.e.n.Node] node name [qYyUHAp] derived from node ID [qYyUHApAREySM8$
> [2017-12-06T10:42:27,367][INFO][o.e.n.Node] version[5.6.3], pid[928], build[1a2f265/2017-10-06T20:33$
> [2017-12-06T10:42:27,368][INFO][o.e.n.Node] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, $
> [2017-12-06T10:42:29,853][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [aggs-matrix-stats]
> [2017-12-06T10:42:29,853][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [ingest-common]
> [2017-12-06T10:42:29,853][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [lang-expression]
> [2017-12-06T10:42:29,853][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [lang-groovy]
> [2017-12-06T10:42:29,853][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [lang-mustache]
> [2017-12-06T10:42:29,854][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [lang-painless]
> [2017-12-06T10:42:29,854][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [parent-join]
> [2017-12-06T10:42:29,854][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [percolator]
> [2017-12-06T10:42:29,854][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [reindex]
> [2017-12-06T10:42:29,854][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [transport-netty3]
> [2017-12-06T10:42:29,854][INFO][o.e.p.PluginsService] [qYyUHAp] loaded module [transport-netty4]
> [2017-12-06T10:42:29,855][INFO][o.e.p.PluginsService] [qYyUHAp] no plugins loaded
> [2017-12-06T10:42:31,396][INFO][o.e.d.DiscoveryModule] [qYyUHAp] using discovery type [zen]
> [2017-12-06T10:42:32,034][INFO][o.e.n.Node] initialized
> [2017-12-06T10:42:32,034][INFO][o.e.n.Node] [qYyUHAp] starting ...
> [2017-12-06T10:42:52,656][INFO][o.e.t.TransportService] [qYyUHAp] publish\_address {127.0.0.1:9300}, bound\_addres$
> [2017-12-06T10:42:55,843][INFO][o.e.c.s.ClusterService] [qYyUHAp] new\_master {qYyUHAp}{qYyUHApAREySM8TuBTwSzg}{q$
> [2017-12-06T10:42:55,873][INFO][o.e.h.n.Netty4HttpServerTransport] [qYyUHAp] publish\_address {127.0.0.1:9200}, boun$
> [2017-12-06T10:42:55,874][INFO][o.e.n.Node] [qYyUHAp] started
> [2017-12-06T10:42:56,246][INFO][o.e.g.GatewayService] [qYyUHAp] recovered [4] indices into cluster\_state
> [2017-12-06T10:43:26,037][WARN][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] high disk watermark [90%] exceeded on [qY$
> [2017-12-06T10:43:26,066][INFO][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] rerouting shards: [high disk watermark ex$
> [2017-12-06T10:43:34,563][INFO][o.e.m.j.JvmGcMonitorService] [qYyUHAp] [gc][62] overhead, spent [377ms] collecting $
> [2017-12-06T10:43:48,695][INFO][o.e.m.j.JvmGcMonitorService] [qYyUHAp] [gc][76] overhead, spent [327ms] collecting $
> [2017-12-06T10:43:56,237][WARN][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] high disk watermark [90%] exceeded on [qY$
Graylog.log on 12/7/2017
> Blockquote
> 2017-12-07T00:00:07,911][WARN][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] high disk watermark [90%] exceeded on [qY$
> [2017-12-07T00:00:07,958][INFO][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] rerouting shards: [high disk watermark ex$
> [2017-12-07T00:00:37,983][WARN][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] high disk watermark [90%] exceeded on [qY$
> [2017-12-07T00:01:08,007][WARN][o.e.c.r.a.DiskThresholdMonitor] [qYyUHAp] high disk watermark [90%] exceeded on [qY$
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 8, 2017, 9:33pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/8 "2017-12-08T21:33:16Z")
> [@dualcore](#):
>
> high disk watermark [90%] exceeded
Maybe try increasing that setting - [Disk-based Shard Allocation | Elasticsearch Reference [6.0] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/disk-allocator.html)
---
### Author:  [@dualcore](https://discuss.elastic.co/u/dualcore)
#### Post date: [December 8, 2017, 9:44pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-corrupted/110721/9 "2017-12-08T21:44:11Z")
To backtrack a little bit. I only have one server running vm 30gb of storage. After 2 weeks the disk was filled. I ran **du -h --max-depth=1** to track the culprit, which lead me to the indices that were deleted because I couldn't eve get the API to launch. After I deleted the files inside the directory suggested above **/var/lib/elasticsearch/**. I rebooted all services and even the server and Elasticsearch still won't start.
When I run curl -X GET [http://localhost:9200](http://localhost:9200). It will spit out:
> Blockquote
> curl -XPOST ‘localhost:9200/\_cluster/reroute’ -d ‘{“commands”:[{“allocate”:{“index”:“graylog2\_6”,“shard”:1,“node”:“gl-es01-esgl2”,“allow\_primary”:true}}]}’
> Access Denied
> Access Denied (authentication\_failed)
> Your credentials could not be authenticated: "Credentials are missing.". You will not be permitted access until your credentials can be verified.
> This is typically caused by an incorrect username and/or password, but could also be caused by network problems.
> Client Name:
> Client IP Address: 192.168.1.10
> Server Name: Host Name
> For assistance, please open a Ticket incident at \