Hi there,
After experiencing some disk usage issues i freed some disk space deleting an old index from Kibana WEBUI interface. I wanted to share the path to another partition with more disk space and proceeded to restart the elasticsearch service. After that the service is not able to start. So I've restored the config back to the original on elasticsearch.yml:
path.data: /var/lib/elasticsearch
The logs thrown by the elasticsearch process are these ones:
[n4gu4l@HOST ~]# tail -f /var/log/elasticsearch/elasticsearch.log | grep "2020-07-09"
[2020-07-09T14:22:50,761][INFO ][o.e.t.TransportService ] [HOST] publish_address {172.16.103.79:9300}, bound_addresses {[::]:9300}
[2020-07-09T14:22:51,268][INFO ][o.e.b.BootstrapChecks ] [HOST] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-07-09T14:22:51,288][ERROR][o.e.b.Bootstrap ] [HOST] node validation exception
[2020-07-09T14:22:51,328][INFO ][o.e.n.Node ] [HOST] stopping ...
[2020-07-09T14:22:51,378][INFO ][o.e.n.Node ] [HOST] stopped
[2020-07-09T14:22:51,378][INFO ][o.e.n.Node ] [HOST] closing ...
[2020-07-09T14:22:51,412][INFO ][o.e.n.Node ] [HOST] closed
[2020-07-09T14:22:51,426][INFO ][o.e.x.m.p.NativeController] [HOST] Native controller process has stopped - no new native processes can be started
[2020-07-09T14:56:24,572][INFO ][o.e.n.Node ] [HOST] version[7.8.0], pid[2124], build[default/rpm/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/4.18.0-193.6.3.el8_2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-07-09T14:56:24,576][INFO ][o.e.n.Node ] [HOST] JVM home [/usr/share/elasticsearch/jdk]
[2020-07-09T14:56:24,576][INFO ][o.e.n.Node ] [HOST] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-11734107667018445712, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-07-09T14:56:26,391][INFO ][o.e.p.PluginsService ] [HOST] loaded module [aggs-matrix-stats]
[2020-07-09T14:56:26,391][INFO ][o.e.p.PluginsService ] [HOST] loaded module [analysis-common]
[2020-07-09T14:56:26,391][INFO ][o.e.p.PluginsService ] [HOST] loaded module [constant-keyword]
[2020-07-09T14:56:26,391][INFO ][o.e.p.PluginsService ] [HOST] loaded module [flattened]
[2020-07-09T14:56:26,391][INFO ][o.e.p.PluginsService ] [HOST] loaded module [frozen-indices]
[2020-07-09T14:56:26,392][INFO ][o.e.p.PluginsService ] [HOST] loaded module [ingest-common]
[2020-07-09T14:56:26,392][INFO ][o.e.p.PluginsService ] [HOST] loaded module [ingest-geoip]
[2020-07-09T14:56:26,392][INFO ][o.e.p.PluginsService ] [HOST] loaded module [ingest-user-agent]
[2020-07-09T14:56:26,392][INFO ][o.e.p.PluginsService ] [HOST] loaded module [kibana]
[2020-07-09T14:56:26,392][INFO ][o.e.p.PluginsService ] [HOST] loaded module [lang-expression]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [lang-mustache]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [lang-painless]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [mapper-extras]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [parent-join]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [percolator]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [rank-eval]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [reindex]
[2020-07-09T14:56:26,393][INFO ][o.e.p.PluginsService ] [HOST] loaded module [repository-url]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [search-business-rules]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [searchable-snapshots]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [spatial]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [systemd]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [tasks]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [transform]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [transport-netty4]
[2020-07-09T14:56:26,394][INFO ][o.e.p.PluginsService ] [HOST] loaded module [vectors]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-analytics]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-async-search]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-autoscaling]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-ccr]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-core]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-deprecation]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-enrich]
[2020-07-09T14:56:26,395][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-eql]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-graph]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-identity-provider]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-ilm]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-logstash]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-ml]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-monitoring]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-ql]
[2020-07-09T14:56:26,396][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-rollup]
[2020-07-09T14:56:26,397][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-security]
[2020-07-09T14:56:26,397][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-sql]
[2020-07-09T14:56:26,397][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-voting-only-node]
[2020-07-09T14:56:26,397][INFO ][o.e.p.PluginsService ] [HOST] loaded module [x-pack-watcher]
[2020-07-09T14:56:26,397][INFO ][o.e.p.PluginsService ] [HOST] no plugins loaded
[2020-07-09T14:56:26,433][INFO ][o.e.e.NodeEnvironment ] [HOST] using [1] data paths, mounts [[/ (/dev/mapper/cl_gm--oss--log--01-root)]], net usable_space [13.4gb], net total_space [49.9gb], types [xfs]
[2020-07-09T14:56:26,434][INFO ][o.e.e.NodeEnvironment ] [HOST] heap size [1gb], compressed ordinary object pointers [true]
[2020-07-09T14:56:26,520][INFO ][o.e.n.Node ] [HOST] node name [HOST], node ID [xBG29k1CQZ-uy9qhC2M6Pg], cluster name [elasticsearch]
[2020-07-09T14:56:29,946][INFO ][o.e.x.s.a.s.FileRolesStore] [HOST] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-07-09T14:56:30,381][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [HOST] [controller/2313] [Main.cc@110] controller (64 bit): Version 7.8.0 (Build 58ff6912e20047) Copyright (c) 2020 Elasticsearch BV
[2020-07-09T14:56:30,973][INFO ][o.e.d.DiscoveryModule ] [HOST] using discovery type [zen] and seed hosts providers [settings]
[2020-07-09T14:56:31,684][INFO ][o.e.n.Node ] [HOST] initialized
[2020-07-09T14:56:31,684][INFO ][o.e.n.Node ] [HOST] starting ...
[2020-07-09T14:56:31,810][INFO ][o.e.t.TransportService ] [HOST] publish_address {172.16.103.79:9300}, bound_addresses {[::]:9300}
[2020-07-09T14:56:32,196][INFO ][o.e.b.BootstrapChecks ] [HOST] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-07-09T14:56:32,217][ERROR][o.e.b.Bootstrap ] [HOST] node validation exception
[2020-07-09T14:56:32,222][INFO ][o.e.n.Node ] [HOST] stopping ...
[2020-07-09T14:56:32,233][INFO ][o.e.n.Node ] [HOST] stopped
[2020-07-09T14:56:32,234][INFO ][o.e.n.Node ] [HOST] closing ...
[2020-07-09T14:56:32,246][INFO ][o.e.n.Node ] [HOST] closed
I've checked this link:
https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html#dev-vs-prod-mode
The deployment mode is single node setup for testing purporses. Not sure why the bootstrap process is failing.
If anybody could shed some light it would be appreaciatted.
Thanks in advance