# 2.x cluster upgraded to 5.x won't start - "failed to read local state, exiting..." on all nodes
**URL:** https://discuss.elastic.co/t/2-x-cluster-upgraded-to-5-x-wont-start-failed-to-read-local-state-exiting-on-all-nodes/154865
**Category:** Elasticsearch
**Created:** [October 31, 2018, 2:35pm UTC](https://discuss.elastic.co/t/2-x-cluster-upgraded-to-5-x-wont-start-failed-to-read-local-state-exiting-on-all-nodes/154865 "2018-10-31T14:35:07Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@vidarno](https://discuss.elastic.co/u/vidarno)
#### Post date: [October 31, 2018, 2:35pm UTC](https://discuss.elastic.co/t/2-x-cluster-upgraded-to-5-x-wont-start-failed-to-read-local-state-exiting-on-all-nodes/154865/1 "2018-10-31T14:35:07Z")
Upgraded a 4-node Elasticsearch 2.x cluster to 5.6.12 - after rebooting all the nodes I get
the following similar log-entries on all nodes;
[2018-10-31T14:59:39,530][WARN][o.e.c.l.LogConfigurator] ignoring unsupported logging configuration file [/etc/elasticsearch/qa-logs01/logging.yml], logging is configured via [/etc/elasticsearch/qa-logs01/log4j2.properties]
[2018-10-31T14:59:39,865][INFO][o.e.n.Node] [logsserver02] initializing ...
[2018-10-31T14:59:40,034][INFO][o.e.e.NodeEnvironment] [logsserver02] using [1] data paths, mounts [[/opt/elasticsearch (192.168.0.10:/qa\_logs01)]], net usable\_space [356.5gb], net total\_space [1.3tb], spins? [possibly], types [nfs4]
[2018-10-31T14:59:40,034][INFO][o.e.e.NodeEnvironment] [logsserver02] heap size [5.9gb], compressed ordinary object pointers [true]
[2018-10-31T15:00:41,003][INFO][o.e.n.Node] [logsserver02] node name [logsserver02], node ID [QL3PAqNbRfukDrgXgnfdCg]
[2018-10-31T15:00:41,007][INFO][o.e.n.Node] [logsserver02] version[5.6.12], pid[13340], build[cfe3d9f/2018-09-10T20:12:43.732Z], OS[Linux/4.4.0-138-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_181/25.181-b13]
[2018-10-31T15:00:41,007][INFO][o.e.n.Node] [logsserver02] JVM arguments [-Dfile.encoding=UTF-8, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djna.nosys=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+AlwaysPreTouch, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintGCDateStamps, -XX:+PrintGCDetails, -XX:+PrintTenuringDistribution, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseConcMarkSweepGC, -XX:+UseGCLogFileRotation, -XX:-OmitStackTraceInFastThrow, -XX:CMSInitiatingOccupancyFraction=75, -XX:GCLogFileSize=64m, -XX:NumberOfGCLogFiles=32, -Xloggc:/var/log/elasticsearch/qa-logs01/gc.log, -Xms2g, -Xmx2g, -Xss1m, -Xms6g, -Xmx6g, -Dmapper.allow\_dots\_in\_name=true, -Des.path.home=/usr/share/elasticsearch]
[2018-10-31T15:00:42,551][INFO][o.e.p.PluginsService] [logsserver02] loaded module [aggs-matrix-stats]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [ingest-common]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [lang-expression]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [lang-groovy]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [lang-mustache]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [lang-painless]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [parent-join]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [percolator]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [reindex]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [transport-netty3]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] loaded module [transport-netty4]
[2018-10-31T15:00:42,552][INFO][o.e.p.PluginsService] [logsserver02] no plugins loaded
[2018-10-31T15:00:45,281][INFO][o.e.d.DiscoveryModule] [logsserver02] using discovery type [zen]
[2018-10-31T15:00:48,495][INFO][o.e.c.u.IndexFolderUpgrader] [kibana-int/jlT5y5OgSh-Y\_1Js8Rei0g] upgrading [/opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int] to new naming convention
[2018-10-31T15:00:48,498][ERROR][o.e.g.GatewayMetaState] [logsserver02] failed to read local state, exiting...
java.nio.file.FileSystemException: /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int -\> /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/jlT5y5OgSh-Y\_1Js8Rei0g: Directory not empty
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
and lines like
[2018-10-31T15:00:51,137][INFO][o.e.c.u.IndexFolderUpgrader] [kibana-int/jlT5y5OgSh-Y\_1Js8Rei0g] upgrading [/opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int] to new naming convention
[2018-10-31T15:00:51,139][ERROR][o.e.g.GatewayMetaState] [logsserver02] failed to read local state, exiting...
java.nio.file.FileSystemException: /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int -\> /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/jlT5y5OgSh-Y\_1Js8Rei0g: Directory not empty
go on a few times before the process dies.
Is there anything I can do to get the cluster up and runnning without losing all my data?
---
### Author:  [@s1monw](https://discuss.elastic.co/u/s1monw)
#### Post date: [November 5, 2018, 1:41pm UTC](https://discuss.elastic.co/t/2-x-cluster-upgraded-to-5-x-wont-start-failed-to-read-local-state-exiting-on-all-nodes/154865/2 "2018-11-05T13:41:44Z")
have you tried this a couple of times and were the error messages the same all the time? It looks like you upgraded half way and then something happened now you are in a limbo state with this node / cluster?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 3, 2018, 1:41pm UTC](https://discuss.elastic.co/t/2-x-cluster-upgraded-to-5-x-wont-start-failed-to-read-local-state-exiting-on-all-nodes/154865/3 "2018-12-03T13:41:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.