Elasticsearch can not started:failed to find global metadata

After the master node is restarted, it cannot be started. The following errors are always reported in the log,Is there any way to enable the Elasticsearch instance to load metadata?

error log:

[2021-10-19T02:32:17,262][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-watcher]
[2021-10-19T02:32:17,263][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] no plugins loaded
[2021-10-19T02:32:21,999][INFO ][o.e.x.s.a.s.FileRolesStore] [tyrzela01-01] parsed [0] roles from file [/ela/ela01/elasticsearch-7.2.0/config/roles.yml]
[2021-10-19T02:32:22,878][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [tyrzela01-01] [controller/20873] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV
[2021-10-19T02:32:23,368][DEBUG][o.e.a.ActionModule       ] [tyrzela01-01] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2021-10-19T02:32:23,529][ERROR][o.e.g.GatewayMetaState   ] [tyrzela01-01] failed to read or upgrade local state, exiting...
java.io.IOException: failed to find global metadata [generation: 5981]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:473) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
[2021-10-19T02:32:23,539][ERROR][o.e.b.Bootstrap          ] [tyrzela01-01] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
Caused by: java.io.IOException: failed to find global metadata [generation: 5981]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:473) ~[elasticsearch-7.2.0.jar:7.2.0]
        ... 11 more

@Mark_Harwood @warkolm

Please don't ping people that aren't already part of the topic :slight_smile:

Are you using docker?
What does your config look like.

I not use docker,Deployed on X86 physical machines

The configuration is as follows:

cluster.name: tyrzelacluster
node.name: tyrzela01-01
node.master: true
node.data: false
path.data:  /data1
path.logs: /ela/ela01/log
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.bind_host: 0.0.0.0
network.publish_host: 152.56.158.142
http.port: 9200
transport.tcp.port: 9300
discovery.seed_hosts: ["152.56.158.142:9300","152.56.158.143:9300","152.56.158.144:9300"]
cluster.initial_master_nodes: ["152.56.158.142","152.56.158.143","152.56.158.144"]
#thread_pool.write.queue_size: 1500
indices.breaker.total.use_real_memory: false
cluster.max_shards_per_node: 2000
#xpack.security.enabled: true
#discovery.zen.fd.ping_timeout: 60s
#discovery.zen.fd.ping_interval: 10s

Don't use both of these, just set network.host.

This is not a great idea, it'll likely lead to inefficient memory use.

What do the Elasticsearch logs show, from startup to this error?

I didn't find any other errors
Here are all the logs:

[2021-10-19T02:32:11,692][INFO ][o.e.e.NodeEnvironment    ] [tyrzela01-01] using [1] data paths, mounts [[/data1 (/dev/sdb)]], net usable_space [11.9tb], net total_space [11.9tb], types [xfs]
[2021-10-19T02:32:11,695][INFO ][o.e.e.NodeEnvironment    ] [tyrzela01-01] heap size [30gb], compressed ordinary object pointers [true]
[2021-10-19T02:32:14,974][INFO ][o.e.n.Node               ] [tyrzela01-01] node name [tyrzela01-01], node ID [-EgjEWF3RSqE8_ZjlwDALg], cluster name [tyrzelacluster]
[2021-10-19T02:32:14,977][INFO ][o.e.n.Node               ] [tyrzela01-01] version[7.2.0], pid[19680], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/3.10.0-862.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2021-10-19T02:32:14,977][INFO ][o.e.n.Node               ] [tyrzela01-01] JVM home [/ela/ela01/elasticsearch-7.2.0/jdk]
[2021-10-19T02:32:14,978][INFO ][o.e.n.Node               ] [tyrzela01-01] JVM arguments [-Xms30g, -Xmx30g, -XX:+UseG1GC, -XX:MaxGCPauseMillis=200, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7204514376905938525, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=16106127360, -Des.path.home=/ela/ela01/elasticsearch-7.2.0, -Des.path.conf=/ela/ela01/elasticsearch-7.2.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-10-19T02:32:17,255][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [aggs-matrix-stats]
[2021-10-19T02:32:17,255][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [analysis-common]
[2021-10-19T02:32:17,256][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [data-frame]
[2021-10-19T02:32:17,256][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [ingest-common]
[2021-10-19T02:32:17,256][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [ingest-geoip]
[2021-10-19T02:32:17,256][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [ingest-user-agent]
[2021-10-19T02:32:17,257][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [lang-expression]
[2021-10-19T02:32:17,257][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [lang-mustache]
[2021-10-19T02:32:17,257][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [lang-painless]
[2021-10-19T02:32:17,258][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [mapper-extras]
[2021-10-19T02:32:17,258][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [parent-join]
[2021-10-19T02:32:17,258][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [percolator]
[2021-10-19T02:32:17,258][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [rank-eval]
[2021-10-19T02:32:17,259][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [reindex]
[2021-10-19T02:32:17,259][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [repository-url]
[2021-10-19T02:32:17,259][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [transport-netty4]
[2021-10-19T02:32:17,259][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-ccr]
[2021-10-19T02:32:17,260][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-core]
[2021-10-19T02:32:17,260][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-deprecation]
[2021-10-19T02:32:17,260][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-graph]
[2021-10-19T02:32:17,260][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-ilm]
[2021-10-19T02:32:17,261][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-logstash]
[2021-10-19T02:32:17,261][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-ml]
[2021-10-19T02:32:17,261][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-monitoring]
[2021-10-19T02:32:17,261][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-rollup]
[2021-10-19T02:32:17,262][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-security]
[2021-10-19T02:32:17,262][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-sql]
[2021-10-19T02:32:17,262][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] loaded module [x-pack-watcher]
[2021-10-19T02:32:17,263][INFO ][o.e.p.PluginsService     ] [tyrzela01-01] no plugins loaded
[2021-10-19T02:32:21,999][INFO ][o.e.x.s.a.s.FileRolesStore] [tyrzela01-01] parsed [0] roles from file [/ela/ela01/elasticsearch-7.2.0/config/roles.yml]
[2021-10-19T02:32:22,878][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [tyrzela01-01] [controller/20873] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV
[2021-10-19T02:32:23,368][DEBUG][o.e.a.ActionModule       ] [tyrzela01-01] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2021-10-19T02:32:23,529][ERROR][o.e.g.GatewayMetaState   ] [tyrzela01-01] failed to read or upgrade local state, exiting...
java.io.IOException: failed to find global metadata [generation: 5981]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:473) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
[2021-10-19T02:32:23,539][ERROR][o.e.b.Bootstrap          ] [tyrzela01-01] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
Caused by: java.io.IOException: failed to find global metadata [generation: 5981]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:473) ~[elasticsearch-7.2.0.jar:7.2.0]
        ... 11 more
[2021-10-19T02:32:23,543][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [tyrzela01-01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to find global metadata [generation: 5981]];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
        ... 6 more
Caused by: java.io.IOException: failed to find global metadata [generation: 5981]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:473) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
        ... 6 more

Please upgrade, 7.2 is EOL and no longer supported.

I tried to start the instance under version 7.4.1 today, but I also reported the same error。
Can I try starting with 7.7 or later?

7.7 is the oldest currently supported version, however 7.15 is latest and recommended.

thanks very much,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.