# Elastic search couldnt start after deleting the nodes-HELP! URGENT **URL:** \ **Category:** Elasticsearch\ **Created:** [October 16, 2019, 4:54am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741 "2019-10-16T04:54:04Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 4:54am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/1 "2019-10-16T04:54:04Z")
# systemctl status elasticsearch รข elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2019-10-16 10:10:43 IST; 12min ago Docs: [http://www.elastic.co](http://www.elastic.co) Process: 6504 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID\_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 6504 (code=exited, status=1/FAILURE) Oct 16 10:10:25 localhost.localdomain systemd[1]: Starting Elasticsearch... Oct 16 10:10:27 localhost.localdomain elasticsearch[6504]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Oct 16 10:10:42 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE Oct 16 10:10:43 localhost.localdomain systemd[1]: Failed to start Elasticsearch. Oct 16 10:10:43 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state. Oct 16 10:10:43 localhost.localdomain systemd[1]: elasticsearch.service failed. Havent change the config file but the indices for elasticsearch is filled completly and there is no server space. Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 14G 9.0G 4.8G 66% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 1.7G 14G 11% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda1 997M 162M 836M 17% /boot /dev/mapper/vgkibana-lvelastic 50G 33M 50G 1% /var/lib/elasticsearch tmpfs 3.2G 0 3.2G 0% /run/user/0 ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [October 16, 2019, 5:18am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/2 "2019-10-16T05:18:38Z")
@anusree_arun, > [@anusree\_arun](#): > > Havent change the config file but the indices for elasticsearch is filled completly and there is no server space. This is the issue. First you need to create some free space on your server then you can restart elasticsaerch service. Thanks. ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 5:23am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/3 "2019-10-16T05:23:16Z")
We have deleted some indices in /var/lib/elasticsearch/nodes/0/\* .After that elastic search cant be started ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [October 16, 2019, 5:28am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/4 "2019-10-16T05:28:26Z")
@anusree_arun, > [@anusree\_arun](#): > > We have deleted some indices in /var/lib/elasticsearch/nodes/0/\* .After that Elasticsearch cant be started Can you please provide some error log? Because error logs may help to fix the issue. Please don't provide service status because they don't have enough info about the issue. Thanks. ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 5:37am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/5 "2019-10-16T05:37:45Z")
**elastic search error log:: attached** [2019-10-16T11:05:53,141][INFO][o.e.e.NodeEnvironment] [localhost.localdomain] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vgkibana-lvelastic)]], net usable\_space [49.9gb], net total\_space [49.9gb], types [xfs] [2019-10-16T11:05:53,177][INFO][o.e.e.NodeEnvironment] [localhost.localdomain] heap size [1007.3mb], compressed ordinary object pointers [true] [2019-10-16T11:05:53,230][INFO][o.e.n.Node] [localhost.localdomain] node name [localhost.localdomain], node ID [KnZ9WJSCSFyrHTeJYwjG9w], cluster name [elasticsearch] [2019-10-16T11:05:53,230][INFO][o.e.n.Node] [localhost.localdomain] version[7.4.0], pid[10014], build[default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z], OS[Linux/3.10.0-957.1.3.el7.x86\_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33] [2019-10-16T11:05:53,231][INFO][o.e.n.Node] [localhost.localdomain] JVM home [/usr/share/elasticsearch/jdk] [2019-10-16T11:05:53,231][INFO][o.e.n.Node] [localhost.localdomain] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-4392861000360210467, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs\_err\_pid%p.log, -Xlog:gc\*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled\_jdk=true] [2019-10-16T11:06:00,426][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [aggs-matrix-stats] [2019-10-16T11:06:00,427][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [analysis-common] [2019-10-16T11:06:00,428][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [data-frame] [2019-10-16T11:06:00,428][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [flattened] [2019-10-16T11:06:00,429][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [frozen-indices] .................. ..................... [2019-10-16T11:06:00,442][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [x-pack-security] [2019-10-16T11:06:00,443][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [x-pack-sql] [2019-10-16T11:06:00,443][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [x-pack-voting-only-node] [2019-10-16T11:06:00,444][INFO][o.e.p.PluginsService] [localhost.localdomain] loaded module [x-pack-watcher] [2019-10-16T11:06:00,445][INFO][o.e.p.PluginsService] [localhost.localdomain] no plugins loaded [2019-10-16T11:06:06,254][INFO][o.e.x.s.a.s.FileRolesStore] [localhost.localdomain] parsed [0] roles from file [/etc/elasticsearch/roles.yml] [2019-10-16T11:06:08,036][INFO][o.e.x.m.p.l.CppLogMessageHandler] [localhost.localdomain] [controller/10111] [Main.cc@110] controller (64 bit): Version 7.4.0 (Build 11d694e7bae395) Copyright (c) 2019 Elasticsearch BV [2019-10-16T11:06:08,818][DEBUG][o.e.a.ActionModule] [localhost.localdomain] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security [2019-10-16T11:06:09,480][ERROR][o.e.g.GatewayMetaState] [localhost.localdomain] failed to read or upgrade local state, exiting... java.io.IOException: failed to find metadata for existing index .watcher-history-10-2019.10.09 [location: u\_Qz9MIcTUS-r5QtdXFQXg, generation: 102] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.4.0.jar:7.4.0] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:141) [elasticsearch-7.4.0.jar:7.4.0] at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:95) [elasticsearch-7.4.0.jar:7.4.0] at org.elasticsearch.node.Node.(Node.java:485) [elasticsearch-7.4.0.jar:7.4.0] at org.elasticsearch.node.Node.(Node.java:255) [elasticsearch-7.4.0.jar:7.4.0] ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 16, 2019, 7:19am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/6 "2019-10-16T07:19:22Z")
> [@anusree\_arun](#): > > We have deleted some indices in /var/lib/elasticsearch/nodes/0/\* .After that Elasticsearch cant be started Unfortunately this will have left this node in a broken state. There are no user-serviceable parts inside the data path and you should never make any changes to it yourself. The only sensible path forwards is to wipe this node. This will allow it to start, and then Elasticsearch will recover the replicas from the other nodes in the cluster. ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 7:19am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/7 "2019-10-16T07:19:58Z")
issue resolved ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 16, 2019, 7:32am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/8 "2019-10-16T07:32:41Z")
Great! I recommend looking further into how this node got so full. By default Elasticsearch will [take action to avoid filling up its disk](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html) and as a last resort will enter read-only mode when the disk reaches 95%. It looks like this didn't happen in your case. Are these protections disabled on your cluster? ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 7:39am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/9 "2019-10-16T07:39:18Z")
i couldn't find the Disk-based shard allocation in my Elasticsearch.yml file ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [October 16, 2019, 7:42am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/10 "2019-10-16T07:42:35Z")
> [@anusree\_arun](#): > > i couldn't find the Disk-based shard allocation in my Elasticsearch.yml file ```auto path.data: /var/lib/elasticsearch ``` this part in `elasticsaerch.yml` file specify where your data will store. ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 7:44am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/11 "2019-10-16T07:44:11Z")
it is not avaialbel in the yml file ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 16, 2019, 7:50am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/12 "2019-10-16T07:50:57Z")
> [@Tek\_Chand](#): > > this part in `elasticsaerch.yml` file specify where your data will store. This isn't relevant. > [@anusree\_arun](#): > > it is not avaialbel in the yml file Is it set as a cluster setting instead? I.e. if you call `GET _cluster/settings` is there any mention of disk watermarks? ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 7:54am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/13 "2019-10-16T07:54:04Z")
{ "persistent" : { "indices" : { "recovery" : { "max\_bytes\_per\_sec" : "50mb" } }, "xpack" : { "monitoring" : { "collection" : { "enabled" : "true" } } } }, "transient" : { } } ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [October 16, 2019, 7:55am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/14 "2019-10-16T07:55:54Z")
> [@DavidTurner](#): > > This isn't relevant. yes..i was thinking something different. She was asking about the setting about watermark the disk if it utilization goes high. Its default setting. Thank you for correcting me. Thanks. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 16, 2019, 7:59am UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/15 "2019-10-16T07:59:49Z")
Ok, no sign of any adjustments to the disk watermarks there. Are they configured differently on any other nodes (particularly, the master-eligible nodes)? Do you have the logs from the time while the node was filling up? I expect to see messages from the `DiskThresholdMonitor` (on the master node) about this node's disk usage. Are there any? ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 2:16pm UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/16 "2019-10-16T14:16:05Z")
couldnt find anything like that ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 2:17pm UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/17 "2019-10-16T14:17:30Z")
nodes are getting filled in elastic search .so that server space is running out. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 16, 2019, 2:27pm UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/18 "2019-10-16T14:27:32Z")
> [@anusree\_arun](#): > > nodes are getting filled in Elasticsearch .so that server space is running out. I don't think we've seen any evidence of this yet. Why do you think that your space is running out? ---
**Author:** ![anusree\_arun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anusree_arun/32/55826_2.png) [@anusree\_arun](https://discuss.elastic.co/u/anusree_arun)\ **Post date:** [October 16, 2019, 2:31pm UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/19 "2019-10-16T14:31:53Z")
this is my server utilisation status Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 14G 9.0G 4.8G 66% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 1.7G 14G 11% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda1 997M 162M 836M 17% /boot **/dev/mapper/vgkibana-lvelastic 50G 33M 50G 1% /var/lib/elasticsearch** tmpfs 3.2G 0 3.2G 0% /run/user/0 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 16, 2019, 2:36pm UTC](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741/20 "2019-10-16T14:36:53Z")
I don't understand. You have highlighted a disk which is using just 33MB out of 50GB i.e. it is about 0.07% full. [Next page](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741.md?page=2)