Hi all,
Due to a disk space problem, I've deleted the indexes and now I'm getting the following error:
java.io.IOException: failed to find metadata for existing index .kibana_task_manager [location: Yi_UxiceTwS093xLAQJpzw, generation: 30]
Have you seen this before? Is it possible to recreate it or somehow start the elastic again?
Can you share the whole error message, including the stack trace?
Also, when you say you deleted the indices, what do you mean? Did you use the delete index API?
Hi David,
Regarding the deleting process, I've just went to /data/nodes/0/ and deleted everything.
Following the whole error message, I've only masked the machine name.
[2019-08-06T16:16:52,047][INFO ][o.e.e.NodeEnvironment ] [XXXXXXX] using [1] data paths, mounts [[/opt/openv (/dev/mapper/vg_nbu-lv_app_nbu)]], net usable_space [1.8gb], net total_space [4.9gb], types [xfs]
[2019-08-06T16:16:52,049][INFO ][o.e.e.NodeEnvironment ] [XXXXXXX] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-08-06T16:16:52,064][INFO ][o.e.n.Node ] [XXXXXXX] node name [XXXXXXX], node ID [Z_bVmvuZRoGqFFKfk7sjKw], cluster name [elasticsearch]
[2019-08-06T16:16:52,065][INFO ][o.e.n.Node ] [XXXXXXX] version[7.2.1], pid[19893], build[default/tar/fe6cb20/2019-07-24T17:58:29.979462Z], OS[Linux/3.10.0-957.12.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2019-08-06T16:16:52,065][INFO ][o.e.n.Node ] [XXXXXXX] JVM home [/opt/openv/elk/elasticsearch-7.2.1/jdk]
[2019-08-06T16:16:52,066][INFO ][o.e.n.Node ] [XXXXXXX] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-17237745443083576117, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/opt/openv/elk/elasticsearch-7.2.1, -Des.path.conf=/opt/openv/elk/elasticsearch-7.2.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
...
[2019-08-06T16:16:54,102][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-graph]
[2019-08-06T16:16:54,103][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-ilm]
[2019-08-06T16:16:54,107][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-logstash]
[2019-08-06T16:16:54,107][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-ml]
[2019-08-06T16:16:54,107][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-monitoring]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-rollup]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-security]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-sql]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-watcher]
[2019-08-06T16:16:54,109][INFO ][o.e.p.PluginsService ] [XXXXXXX] no plugins loaded
[2019-08-06T16:16:58,334][INFO ][o.e.x.s.a.s.FileRolesStore] [XXXXXXX] parsed [0] roles from file [/opt/openv/elk/elasticsearch-7.2.1/config/roles.yml]
[2019-08-06T16:16:59,198][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [XXXXXXX] [controller/19992] [Main.cc@110] controller (64 bit): Version 7.2.1 (Build 4ad685337be7fd) Copyright (c) 2019 Elasticsearch BV
[2019-08-06T16:16:59,620][DEBUG][o.e.a.ActionModule ] [XXXXXXX] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-08-06T16:16:59,789][ERROR][o.e.g.GatewayMetaState ] [XXXXXXX] failed to read or upgrade local state, exiting...
java.io.IOException: failed to find metadata for existing index .kibana_task_manager [location: Yi_UxiceTwS093xLAQJpzw, generation: 30]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:102) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:473) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:251) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.1.jar:7.2.1]
[2019-08-06T16:16:59,797][ERROR][o.e.b.Bootstrap ] [XXXXXXX] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
at org.elasticsearch.node.Node.(Node.java:580) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.1.jar:7.2.1]
"logs/elasticsearch.log" 103L, 12907C
Are you sure you deleted everything under /data/nodes/0/? I think you might have left something in /data/nodes/0/_state. You'd be best off deleting the whole 0 folder.
You were absolutely right. After deleting the folder it worked.
Thank you.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.