Elastic search couldnt start after deleting the nodes-HELP! URGENT

Elastic Stack Elasticsearch
1

systemctl status elasticsearch

â elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2019-10-16 10:10:43 IST; 12min ago
Docs: http://www.elastic.co
Process: 6504 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 6504 (code=exited, status=1/FAILURE)

Oct 16 10:10:25 localhost.localdomain systemd[1]: Starting Elasticsearch...
Oct 16 10:10:27 localhost.localdomain elasticsearch[6504]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Oct 16 10:10:42 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 16 10:10:43 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Oct 16 10:10:43 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Oct 16 10:10:43 localhost.localdomain systemd[1]: elasticsearch.service failed.

Havent change the config file but the indices for elasticsearch is filled completly and there is no server space.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 14G 9.0G 4.8G 66% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.7G 14G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 997M 162M 836M 17% /boot
/dev/mapper/vgkibana-lvelastic 50G 33M 50G 1% /var/lib/elasticsearch
tmpfs 3.2G 0 3.2G 0% /run/user/0

2

@anusree_arun,

This is the issue. First you need to create some free space on your server then you can restart elasticsaerch service.

Thanks.

3

We have deleted some indices in /var/lib/elasticsearch/nodes/0/* .After that elastic search cant be started

4

@anusree_arun,

Can you please provide some error log? Because error logs may help to fix the issue. Please don't provide service status because they don't have enough info about the issue.

Thanks.

5

elastic search error log:: attached
[2019-10-16T11:05:53,141][INFO ][o.e.e.NodeEnvironment ] [localhost.localdomain] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vgkibana-lvelastic)]], net usable_space [49.9gb], net total_space [49.9gb], types [xfs]
[2019-10-16T11:05:53,177][INFO ][o.e.e.NodeEnvironment ] [localhost.localdomain] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-10-16T11:05:53,230][INFO ][o.e.n.Node ] [localhost.localdomain] node name [localhost.localdomain], node ID [KnZ9WJSCSFyrHTeJYwjG9w], cluster name [elasticsearch]
[2019-10-16T11:05:53,230][INFO ][o.e.n.Node ] [localhost.localdomain] version[7.4.0], pid[10014], build[default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z], OS[Linux/3.10.0-957.1.3.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]
[2019-10-16T11:05:53,231][INFO ][o.e.n.Node ] [localhost.localdomain] JVM home [/usr/share/elasticsearch/jdk]
[2019-10-16T11:05:53,231][INFO ][o.e.n.Node ] [localhost.localdomain] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-4392861000360210467, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2019-10-16T11:06:00,426][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [aggs-matrix-stats]
[2019-10-16T11:06:00,427][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [analysis-common]
[2019-10-16T11:06:00,428][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [data-frame]
[2019-10-16T11:06:00,428][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [flattened]
[2019-10-16T11:06:00,429][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [frozen-indices]
..................

.....................

[2019-10-16T11:06:00,442][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [x-pack-security]
[2019-10-16T11:06:00,443][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [x-pack-sql]
[2019-10-16T11:06:00,443][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [x-pack-voting-only-node]
[2019-10-16T11:06:00,444][INFO ][o.e.p.PluginsService ] [localhost.localdomain] loaded module [x-pack-watcher]
[2019-10-16T11:06:00,445][INFO ][o.e.p.PluginsService ] [localhost.localdomain] no plugins loaded
[2019-10-16T11:06:06,254][INFO ][o.e.x.s.a.s.FileRolesStore] [localhost.localdomain] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-10-16T11:06:08,036][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [localhost.localdomain] [controller/10111] [Main.cc@110] controller (64 bit): Version 7.4.0 (Build 11d694e7bae395) Copyright (c) 2019 Elasticsearch BV
[2019-10-16T11:06:08,818][DEBUG][o.e.a.ActionModule ] [localhost.localdomain] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-10-16T11:06:09,480][ERROR][o.e.g.GatewayMetaState ] [localhost.localdomain] failed to read or upgrade local state, exiting...
java.io.IOException: failed to find metadata for existing index .watcher-history-10-2019.10.09 [location: u_Qz9MIcTUS-r5QtdXFQXg, generation: 102]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:141) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:95) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.node.Node.(Node.java:485) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.node.Node.(Node.java:255) [elasticsearch-7.4.0.jar:7.4.0]

6

Unfortunately this will have left this node in a broken state. There are no user-serviceable parts inside the data path and you should never make any changes to it yourself.

The only sensible path forwards is to wipe this node. This will allow it to start, and then Elasticsearch will recover the replicas from the other nodes in the cluster.

7

issue resolved

8

Great!

I recommend looking further into how this node got so full. By default Elasticsearch will take action to avoid filling up its disk and as a last resort will enter read-only mode when the disk reaches 95%. It looks like this didn't happen in your case. Are these protections disabled on your cluster?

9

i couldn't find the Disk-based shard allocation in my Elasticsearch.yml file

10 
path.data: /var/lib/elasticsearch

this part in elasticsaerch.yml file specify where your data will store.

11

it is not avaialbel in the yml file

12

This isn't relevant.

Is it set as a cluster setting instead? I.e. if you call GET _cluster/settings is there any mention of disk watermarks?

13

{
"persistent" : {
"indices" : {
"recovery" : {
"max_bytes_per_sec" : "50mb"
}
},
"xpack" : {
"monitoring" : {
"collection" : {
"enabled" : "true"
}
}
}
},
"transient" : { }
}

14

yes..i was thinking something different. She was asking about the setting about watermark the disk if it utilization goes high. Its default setting.

Thank you for correcting me.

Thanks.

15

Ok, no sign of any adjustments to the disk watermarks there. Are they configured differently on any other nodes (particularly, the master-eligible nodes)?

Do you have the logs from the time while the node was filling up? I expect to see messages from the DiskThresholdMonitor (on the master node) about this node's disk usage. Are there any?

16

couldnt find anything like that

17

nodes are getting filled in elastic search .so that server space is running out.

18

I don't think we've seen any evidence of this yet. Why do you think that your space is running out?

19

this is my server utilisation status

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 14G 9.0G 4.8G 66% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.7G 14G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 997M 162M 836M 17% /boot
/dev/mapper/vgkibana-lvelastic 50G 33M 50G 1% /var/lib/elasticsearch
tmpfs 3.2G 0 3.2G 0% /run/user/0

20

I don't understand. You have highlighted a disk which is using just 33MB out of 50GB i.e. it is about 0.07% full.