Elasticsearch says it fails to start, but actually starts up

bennett · June 5, 2017, 6:45pm

Hi,

I'm trying to figure out why when i restart the elasticsearch service, it says in the console it fails to start, but in the logs, there are no errors and it starts up. When I put the log in debug mode, i still dont' see any errors. And it actually starts up.

Does anyone know why this happens?

I'm running the Current release 5.4.0

kbennett@gew1-elasticsearch-warm-wmw9:~$ sudo /etc/init.d/elasticsearch-es-weblog restart

Stopping Elasticsearch Server es-weblog [ OK ]
Starting Elasticsearch Server es-weblog [fail]

log:

[2017-06-05T18:31:39,428][INFO ][o.e.n.Node ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] initializing ...
[2017-06-05T18:31:39,522][INFO ][o.e.e.NodeEnvironment ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [3.8tb], net total_space [7.6tb], spins? [possibly], types [ext4]
[2017-06-05T18:31:39,523][INFO ][o.e.e.NodeEnvironment ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] heap size [30.8gb], compressed ordinary object pointers [true]
[2017-06-05T18:31:39,927][INFO ][o.e.n.Node ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] node name [gew1-elasticsearch-warm-wmw9.gew1.domain.net], node ID [FX2iHNMRR7Wf91cqtKI45A]
[2017-06-05T18:31:39,927][INFO ][o.e.n.Node ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] version[5.4.0], pid[12618], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/3.16.0-77-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_102/25.102-b14]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [aggs-matrix-stats]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [ingest-common]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [lang-expression]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [lang-groovy]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [lang-mustache]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [lang-painless]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [percolator]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [reindex]
[2017-06-05T18:31:40,805][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [transport-netty3]
[2017-06-05T18:31:40,806][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] loaded module [transport-netty4]
[2017-06-05T18:31:40,807][INFO ][o.e.p.PluginsService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] no plugins loaded
[2017-06-05T18:31:42,278][INFO ][o.e.d.DiscoveryModule ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] using discovery type [zen]
[2017-06-05T18:31:43,674][INFO ][o.e.n.Node ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] initialized
[2017-06-05T18:31:43,674][INFO ][o.e.n.Node ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] starting ...
[2017-06-05T18:31:44,459][INFO ][o.e.t.TransportService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] publish_address {10.173.12.227:9300}, bound_addresses {[::]:9300}
[2017-06-05T18:31:44,467][INFO ][o.e.b.BootstrapChecks ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-06-05T18:31:48,529][INFO ][o.e.c.s.ClusterService ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] detected_master reason: zen-disco-receive(from master [master {gew1-elasticsearch-master-cpr4.gew1.domain.net}{} committed version [5603]])
[2017-06-05T18:31:48,827][INFO ][o.e.m.j.JvmGcMonitorService] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] [gc][5] overhead, spent [373ms] collecting in the last [1.1s]
[2017-06-05T18:31:49,092][INFO ][o.e.c.s.ClusterSettings ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [20]
[2017-06-05T18:31:49,093][INFO ][o.e.c.s.ClusterSettings ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] updating [cluster.routing.allocation.enable] from [ALL] to [all]
[2017-06-05T18:31:49,093][INFO ][o.e.c.s.ClusterSettings ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] updating [cluster.routing.rebalance.enable] from [ALL] to [none]
[2017-06-05T18:31:49,093][INFO ][o.e.c.s.ClusterSettings ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [20]
[2017-06-05T18:31:49,093][INFO ][o.e.c.s.ClusterSettings ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] updating [cluster.routing.allocation.node_concurrent_outgoing_recoveries] from [2] to [20]
[2017-06-05T18:31:49,093][INFO ][o.e.c.s.ClusterSettings ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] updating [indices.recovery.max_bytes_per_sec] from [40mb] to [200mb]
[2017-06-05T18:31:50,025][INFO ][o.e.h.n.Netty4HttpServerTransport] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] publish_address {:9200}, bound_addresses {[::]:9200}
[2017-06-05T18:31:50,028][INFO ][o.e.n.Node ] [gew1-elasticsearch-warm-wmw9.gew1.domain.net] started

xavierfacq · June 5, 2017, 8:16pm

Hi, Does a stop and then a start fail too ?

bennett · June 5, 2017, 8:18pm

yes, it does

xavierfacq · June 5, 2017, 8:23pm

Can you stop it, delete logs, and start again, then post logs here plz.
Any log in /var/log/messages ?

bennett · June 5, 2017, 9:14pm

I think i got it......

I had to increase the start timeout in the init.d script. and so far it's been "ok"

bennett · June 5, 2017, 10:04pm

althoug i do get a lot of...

[2017-06-05T22:02:08,054][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/spotify/var/elasticsearch/es-weblog]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:260) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.node.Node.(Node.java:262) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.node.Node.(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]
[2017-06-05T22:02:08,060][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gew1-elasticsearch-warm-mk2x.gew1.domain.net] uncaught exception in thread [main]

even after i kill the process and start.

xavierfacq · June 12, 2017, 7:44am

Sorry for the late response, does your directory wriable ?

system · July 10, 2017, 7:44am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.