Upgraded a 4-node Elasticsearch 2.x cluster to 5.6.12 - after rebooting all the nodes I get
the following similar log-entries on all nodes;
[2018-10-31T14:59:39,530][WARN ][o.e.c.l.LogConfigurator ] ignoring unsupported logging configuration file [/etc/elasticsearch/qa-logs01/logging.yml], logging is configured via [/etc/elasticsearch/qa-logs01/log4j2.properties]
[2018-10-31T14:59:39,865][INFO ][o.e.n.Node ] [logsserver02] initializing ...
[2018-10-31T14:59:40,034][INFO ][o.e.e.NodeEnvironment ] [logsserver02] using [1] data paths, mounts [[/opt/elasticsearch (192.168.0.10:/qa_logs01)]], net usable_space [356.5gb], net total_space [1.3tb], spins? [possibly], types [nfs4]
[2018-10-31T14:59:40,034][INFO ][o.e.e.NodeEnvironment ] [logsserver02] heap size [5.9gb], compressed ordinary object pointers [true]
[2018-10-31T15:00:41,003][INFO ][o.e.n.Node ] [logsserver02] node name [logsserver02], node ID [QL3PAqNbRfukDrgXgnfdCg]
[2018-10-31T15:00:41,007][INFO ][o.e.n.Node ] [logsserver02] version[5.6.12], pid[13340], build[cfe3d9f/2018-09-10T20:12:43.732Z], OS[Linux/4.4.0-138-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-10-31T15:00:41,007][INFO ][o.e.n.Node ] [logsserver02] JVM arguments [-Dfile.encoding=UTF-8, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djna.nosys=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+AlwaysPreTouch, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintGCDateStamps, -XX:+PrintGCDetails, -XX:+PrintTenuringDistribution, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseConcMarkSweepGC, -XX:+UseGCLogFileRotation, -XX:-OmitStackTraceInFastThrow, -XX:CMSInitiatingOccupancyFraction=75, -XX:GCLogFileSize=64m, -XX:NumberOfGCLogFiles=32, -Xloggc:/var/log/elasticsearch/qa-logs01/gc.log, -Xms2g, -Xmx2g, -Xss1m, -Xms6g, -Xmx6g, -Dmapper.allow_dots_in_name=true, -Des.path.home=/usr/share/elasticsearch]
[2018-10-31T15:00:42,551][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [aggs-matrix-stats]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [ingest-common]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [lang-expression]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [lang-groovy]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [lang-mustache]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [lang-painless]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [parent-join]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [percolator]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [reindex]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [transport-netty3]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] loaded module [transport-netty4]
[2018-10-31T15:00:42,552][INFO ][o.e.p.PluginsService ] [logsserver02] no plugins loaded
[2018-10-31T15:00:45,281][INFO ][o.e.d.DiscoveryModule ] [logsserver02] using discovery type [zen]
[2018-10-31T15:00:48,495][INFO ][o.e.c.u.IndexFolderUpgrader] [kibana-int/jlT5y5OgSh-Y_1Js8Rei0g] upgrading [/opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int] to new naming convention
[2018-10-31T15:00:48,498][ERROR][o.e.g.GatewayMetaState ] [logsserver02] failed to read local state, exiting...
java.nio.file.FileSystemException: /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int -> /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/jlT5y5OgSh-Y_1Js8Rei0g: Directory not empty
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
and lines like
[2018-10-31T15:00:51,137][INFO ][o.e.c.u.IndexFolderUpgrader] [kibana-int/jlT5y5OgSh-Y_1Js8Rei0g] upgrading [/opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int] to new naming convention
[2018-10-31T15:00:51,139][ERROR][o.e.g.GatewayMetaState ] [logsserver02] failed to read local state, exiting...
java.nio.file.FileSystemException: /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/kibana-int -> /opt/elasticsearch/logsserver02/qa-logs01/nodes/0/indices/jlT5y5OgSh-Y_1Js8Rei0g: Directory not empty
go on a few times before the process dies.
Is there anything I can do to get the cluster up and runnning without losing all my data?