Cant start ES after update from 5.6.2 to 6.0

Elastic Stack Elasticsearch
1

ENV:
OS: Centos 7
ES : 56.2 to 6.0

After upgrade, i reinstall all plugin ES, after that start ES but got error

Nov 23 15:50:33 elastic-04 systemd[1]: Unit elasticsearch.service entered failed state.
Nov 23 15:53:07 elastic-04 systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 23 15:53:07 elastic-04 systemd[5047]: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such
-- Subject: Process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed and failed.
--
-- The error number returned while executing this process is 2.
Nov 23 15:53:07 elastic-04 systemd[1]: elasticsearch.service: control process exited, code=exited status=203
Nov 23 15:53:07 elastic-04 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 23 15:53:07 elastic-04 systemd[1]: Unit elasticsearch.service entered failed state.

And here is message log

Nov 23 16:16:45 Elastic-04 systemd: elasticsearch.service: control process exited, code=exited status=203
Nov 23 16:16:45 Elastic-04 systemd: Failed to start Elasticsearch.
Nov 23 16:16:45 Elastic-04 systemd: Unit elasticsearch.service entered failed state.
Nov 23 16:16:50 Elastic-04 systemd: Starting Elasticsearch...
Nov 23 16:16:50 Elastic-04 systemd: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such file or directory
Nov 23 16:16:50 Elastic-04 systemd: elasticsearch.service: control process exited, code=exited status=203
Nov 23 16:16:50 Elastic-04 systemd: Failed to start Elasticsearch.
Nov 23 16:16:50 Elastic-04 systemd: Unit elasticsearch.service entered failed state

I try to set ES_PATH_CONF with :

export ES_PATH_CONF =/etc/elasticsearch/

But ES cannot start!!! What happen?

P/s: Dont have log in folder log ES

Thansk

2

Can someone help me?
I have 5 node in production, all is down ...

3

use bin/elasticsearch start show the log. not use systemctl.

4

This is the key log line. We do not have an ExecStartPre entry in 6.0.0 yet clearly your unit file still refers to one after the upgrade. This means you still have an old 5.x unit file and the upgrade did not replace the unit file with the 6.0.0 unit file. Maybe you edited the unit file (don’t do that!) and when you upgraded the package installer gave you a .rpmnew. You need to use the 6.0.0 unit file.

Try to test changes to your production system in a staging environment first.

1 Like
5

Yeah, i just found problem,

I used new startup file but i got new error from log.

[2017-11-24T10:27:40,576][INFO ][o.e.n.Node               ] [Elastic-03] initializing ...
[2017-11-24T10:27:40,674][INFO ][o.e.e.NodeEnvironment    ] [Elastic-03] using [1] data paths, mounts [[/data (/dev/sdb1)]], net usable_space [536gb], net total_space [1.8tb], types [ext4]
[2017-11-24T10:27:40,674][INFO ][o.e.e.NodeEnvironment    ] [Elastic-03] heap size [19.8gb], compressed ordinary object pointers [true]
[2017-11-24T10:27:40,936][INFO ][o.e.n.Node               ] [Elastic-03] node name [Elastic-03], node ID [DTbL2JaQQTGTcRPDC5c-iA]
[2017-11-24T10:27:40,936][INFO ][o.e.n.Node               ] [Elastic-03] version[6.0.0], pid[662], build[8f0685b/2017-11-10T18:41:22.859Z], OS[Linux/3.10.0-229.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-11-24T10:27:40,936][INFO ][o.e.n.Node               ] [Elastic-03] JVM arguments [-Xms20g, -Xmx20g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [aggs-matrix-stats]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [analysis-common]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [ingest-common]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [lang-expression]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [lang-mustache]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [lang-painless]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [parent-join]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [percolator]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [reindex]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [repository-url]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [transport-netty4]
[2017-11-24T10:27:42,241][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded module [tribe]
[2017-11-24T10:27:42,242][INFO ][o.e.p.PluginsService     ] [Elastic-03] loaded plugin [x-pack]
[2017-11-24T10:27:43,992][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2017-11-24T10:27:45,076][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/750] [Main.cc@128] controller (64 bit): Version 6.0.0 (Build 8e6ab35cf803a2) Copyright (c) 2017 Elasticsearch BV
[2017-11-24T10:27:45,137][INFO ][o.e.d.DiscoveryModule    ] [Elastic-03] using discovery type [zen]
[2017-11-24T10:27:46,374][ERROR][o.e.g.GatewayMetaState   ] [Elastic-03] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:2992, legacy:false, file:/data/elasticsearch/data/nodes/0/_state/global-2992.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:150) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-6.0.0.jar:6.0.0]
        at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:113) ~[elasticsearch-6.0.0.jar:6.0.0]

I upgrade from 5.6.2 to 6.0

I try to start on two node, same result

6

That is this known issue from ML: Elasticsearch 5.6.3 upgrade to 6.0 Node will not start "failed to read local state"

7

Thanks @jasontedor,

But now cluster is down, so i cant delete Ml or access to them

have another way?

8

You have to roll back to 5.6 and delete the offending ML job. It's a bug, and that's the workaround for now. Sorry that I do not have better news.

1 Like
9

yeah thank @jasontedor, I'm trying fix that, but maybe i lost all data because rollback false.

10

I am not sure what you mean, you should not lose any data here. Usually you can not downgrade, however in this case you can because startup on 6.0.0 never happened successfully.

11

Hi @jasontedor,

I posted some log in this port ... , if you can, help me solve this problem.

Sumary:

  • Upgrade from 5.6.2 to 6.0.0, fail becasue ML job .
  • Downgrade from 6.0 back to 5.6.2 -> delete ML job but cant not start cluster.

I sent log for dkyle . So we can move to thread and find a way solve this problem.
P/s: I sent log for you in private message.

Thanks for all

12

Hi,

Elasticsearch service is different in version 6. If you modified it when you are using version 5.X you might begin not to use it after upgrade. I realized that I changed it for data path in version 5.X, however there is no need to change elastic service for data path in version 6.

So I am sharing version 6 service config below. You can backup your own and replace it with the below and deamon reload and start service may solve your problem.

Regards,

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=elasticsearch
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

StandardOutput=journal
StandardError=inherit
LimitNOFILE=65536
LimitNPROC=4096
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143
WantedBy=multi-user.target

Built for distribution-6.0.0 (distribution)

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.