# Cant start ES after update from 5.6.2 to 6.0
**URL:** https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843
**Category:** Elasticsearch
**Created:** [November 23, 2017, 8:59am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843 "2017-11-23T08:59:43Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 23, 2017, 8:59am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/1 "2017-11-23T08:59:43Z")
ENV:
OS: Centos 7
ES : 56.2 to 6.0
After upgrade, i reinstall all plugin ES, after that start ES but got error
```
Nov 23 15:50:33 elastic-04 systemd[1]: Unit elasticsearch.service entered failed state.
Nov 23 15:53:07 elastic-04 systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 23 15:53:07 elastic-04 systemd[5047]: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such
-- Subject: Process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec could not be executed and failed.
--
-- The error number returned while executing this process is 2.
Nov 23 15:53:07 elastic-04 systemd[1]: elasticsearch.service: control process exited, code=exited status=203
Nov 23 15:53:07 elastic-04 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 23 15:53:07 elastic-04 systemd[1]: Unit elasticsearch.service entered failed state.
```
And here is message log
```
Nov 23 16:16:45 Elastic-04 systemd: elasticsearch.service: control process exited, code=exited status=203
Nov 23 16:16:45 Elastic-04 systemd: Failed to start Elasticsearch.
Nov 23 16:16:45 Elastic-04 systemd: Unit elasticsearch.service entered failed state.
Nov 23 16:16:50 Elastic-04 systemd: Starting Elasticsearch...
Nov 23 16:16:50 Elastic-04 systemd: Failed at step EXEC spawning /usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec: No such file or directory
Nov 23 16:16:50 Elastic-04 systemd: elasticsearch.service: control process exited, code=exited status=203
Nov 23 16:16:50 Elastic-04 systemd: Failed to start Elasticsearch.
Nov 23 16:16:50 Elastic-04 systemd: Unit elasticsearch.service entered failed state
```
I try to set ES\_PATH\_CONF with :
`export ES_PATH_CONF =/etc/elasticsearch/`
But ES cannot start!!! What happen?
P/s: Dont have log in folder log ES
Thansk
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 24, 2017, 2:15am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/2 "2017-11-24T02:15:30Z")
Can someone help me?
I have 5 node in production, all is down ...
---
### Author:  [@zqc0512](https://discuss.elastic.co/u/zqc0512)
#### Post date: [November 24, 2017, 2:17am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/3 "2017-11-24T02:17:01Z")
use bin/elasticsearch start show the log. not use systemctl.
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [November 24, 2017, 2:43am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/4 "2017-11-24T02:43:33Z")
> [@tatdat](#):
>
> elasticsearch-systemd-pre-exec could not be executed
This is the key log line. We do not have an `ExecStartPre` entry in 6.0.0 yet clearly your unit file still refers to one after the upgrade. This means you still have an old 5.x unit file and the upgrade did not replace the unit file with the 6.0.0 unit file. Maybe you edited the unit file (don’t do that!) and when you upgraded the package installer gave you a .rpmnew. You need to use the 6.0.0 unit file.
> [@tatdat](#):
>
> I have 5 node in production, all is down ...
Try to test changes to your production system in a staging environment first.
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 24, 2017, 3:35am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/5 "2017-11-24T03:35:38Z")
> [@jasontedor](#):
>
> This is the key log line. We do not have an ExecStartPre entry in 6.0.0 yet clearly your unit file still refers to one after the upgrade. This means you still have an old 5.x unit file and the upgrade did not replace the unit file with the 6.0.0 unit file. Maybe you edited the unit file (don’t do that!) and when you upgraded the package installer gave you a .rpmnew. You need to use the 6.0.0 unit file.
Yeah, i just found problem,
I used new startup file but i got new error from log.
```
[2017-11-24T10:27:40,576][INFO][o.e.n.Node] [Elastic-03] initializing ...
[2017-11-24T10:27:40,674][INFO][o.e.e.NodeEnvironment] [Elastic-03] using [1] data paths, mounts [[/data (/dev/sdb1)]], net usable_space [536gb], net total_space [1.8tb], types [ext4]
[2017-11-24T10:27:40,674][INFO][o.e.e.NodeEnvironment] [Elastic-03] heap size [19.8gb], compressed ordinary object pointers [true]
[2017-11-24T10:27:40,936][INFO][o.e.n.Node] [Elastic-03] node name [Elastic-03], node ID [DTbL2JaQQTGTcRPDC5c-iA]
[2017-11-24T10:27:40,936][INFO][o.e.n.Node] [Elastic-03] version[6.0.0], pid[662], build[8f0685b/2017-11-10T18:41:22.859Z], OS[Linux/3.10.0-229.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-11-24T10:27:40,936][INFO][o.e.n.Node] [Elastic-03] JVM arguments [-Xms20g, -Xmx20g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [aggs-matrix-stats]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [analysis-common]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [ingest-common]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [lang-expression]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [lang-mustache]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [lang-painless]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [parent-join]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [percolator]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [reindex]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [repository-url]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [transport-netty4]
[2017-11-24T10:27:42,241][INFO][o.e.p.PluginsService] [Elastic-03] loaded module [tribe]
[2017-11-24T10:27:42,242][INFO][o.e.p.PluginsService] [Elastic-03] loaded plugin [x-pack]
[2017-11-24T10:27:43,992][DEBUG][o.e.a.ActionModule] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2017-11-24T10:27:45,076][INFO][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/750] [Main.cc@128] controller (64 bit): Version 6.0.0 (Build 8e6ab35cf803a2) Copyright (c) 2017 Elasticsearch BV
[2017-11-24T10:27:45,137][INFO][o.e.d.DiscoveryModule] [Elastic-03] using discovery type [zen]
[2017-11-24T10:27:46,374][ERROR][o.e.g.GatewayMetaState] [Elastic-03] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:2992, legacy:false, file:/data/elasticsearch/data/nodes/0/_state/global-2992.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:150) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:113) ~[elasticsearch-6.0.0.jar:6.0.0]
```
I upgrade from 5.6.2 to 6.0
I try to start on two node, same result
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [November 24, 2017, 5:24am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/6 "2017-11-24T05:24:14Z")
That is this known issue from ML: [Elasticsearch 5.6.3 upgrade to 6.0 Node will not start "failed to read local state"](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252)
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 24, 2017, 6:28am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/7 "2017-11-24T06:28:32Z")
Thanks @jasontedor,
But now cluster is down, so i cant delete Ml or access to them
have another way?
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [November 24, 2017, 10:25pm UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/8 "2017-11-24T22:25:17Z")
You have to roll back to 5.6 and delete the offending ML job. It's a bug, and that's the workaround for now. Sorry that I do not have better news.
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 25, 2017, 10:39am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/9 "2017-11-25T10:39:42Z")
yeah thank @jasontedor, I'm trying fix that, but maybe i lost all data because rollback false.
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [November 26, 2017, 12:23am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/10 "2017-11-26T00:23:18Z")
I am not sure what you mean, you should not lose any data here. Usually you can not downgrade, however in this case you can because startup on 6.0.0 never happened successfully.
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 26, 2017, 2:31pm UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/11 "2017-11-26T14:31:04Z")
Hi @jasontedor,
I posted some log in [this port](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/22)... , if you can, help me solve this problem.
Sumary:
- Upgrade from 5.6.2 to 6.0.0, fail becasue ML job .
- Downgrade from 6.0 back to 5.6.2 -\> delete ML job but cant not start cluster.
I sent log for **dkyle**. So we can move to [thread](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/22) and find a way solve this problem.
P/s: I sent log for you in private message.
Thanks for all
---
### Author:  [@realin](https://discuss.elastic.co/u/realin)
#### Post date: [November 30, 2017, 6:11am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/12 "2017-11-30T06:11:59Z")
Hi,
Elasticsearch service is different in version 6. If you modified it when you are using version 5.X you might begin not to use it after upgrade. I realized that I changed it for data path in version 5.X, however there is no need to change elastic service for data path in version 6.
So I am sharing version 6 service config below. You can backup your own and replace it with the below and deamon reload and start service may solve your problem.
Regards,
[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
RuntimeDirectory=elasticsearch
Environment=ES\_HOME=/usr/share/elasticsearch
Environment=ES\_PATH\_CONF=/etc/elasticsearch
Environment=PID\_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch
WorkingDirectory=/usr/share/elasticsearch
User=elasticsearch
Group=elasticsearch
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID\_DIR}/elasticsearch.pid --quiet
StandardOutput=journal
StandardError=inherit
LimitNOFILE=65536
LimitNPROC=4096
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143
WantedBy=multi-user.target
# Built for distribution-6.0.0 (distribution)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 28, 2017, 6:12am UTC](https://discuss.elastic.co/t/cant-start-es-after-update-from-5-6-2-to-6-0/108843/13 "2017-12-28T06:12:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.