Failed to find metadata for existing index .kibana_task_manager [location: Yi_UxiceTwS093xLAQJpzw, generation: 30]

Elastic Stack Elasticsearch
1

Hi all,
Due to a disk space problem, I've deleted the indexes and now I'm getting the following error:

java.io.IOException: failed to find metadata for existing index .kibana_task_manager [location: Yi_UxiceTwS093xLAQJpzw, generation: 30]

Have you seen this before? Is it possible to recreate it or somehow start the elastic again?

2

Can you share the whole error message, including the stack trace?

Also, when you say you deleted the indices, what do you mean? Did you use the delete index API?

3

Hi David,

Regarding the deleting process, I've just went to /data/nodes/0/ and deleted everything.

Following the whole error message, I've only masked the machine name.
[2019-08-06T16:16:52,047][INFO ][o.e.e.NodeEnvironment ] [XXXXXXX] using [1] data paths, mounts [[/opt/openv (/dev/mapper/vg_nbu-lv_app_nbu)]], net usable_space [1.8gb], net total_space [4.9gb], types [xfs]
[2019-08-06T16:16:52,049][INFO ][o.e.e.NodeEnvironment ] [XXXXXXX] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-08-06T16:16:52,064][INFO ][o.e.n.Node ] [XXXXXXX] node name [XXXXXXX], node ID [Z_bVmvuZRoGqFFKfk7sjKw], cluster name [elasticsearch]
[2019-08-06T16:16:52,065][INFO ][o.e.n.Node ] [XXXXXXX] version[7.2.1], pid[19893], build[default/tar/fe6cb20/2019-07-24T17:58:29.979462Z], OS[Linux/3.10.0-957.12.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2019-08-06T16:16:52,065][INFO ][o.e.n.Node ] [XXXXXXX] JVM home [/opt/openv/elk/elasticsearch-7.2.1/jdk]
[2019-08-06T16:16:52,066][INFO ][o.e.n.Node ] [XXXXXXX] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-17237745443083576117, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/opt/openv/elk/elasticsearch-7.2.1, -Des.path.conf=/opt/openv/elk/elasticsearch-7.2.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
...
[2019-08-06T16:16:54,102][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-graph]
[2019-08-06T16:16:54,103][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-ilm]
[2019-08-06T16:16:54,107][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-logstash]
[2019-08-06T16:16:54,107][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-ml]
[2019-08-06T16:16:54,107][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-monitoring]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-rollup]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-security]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-sql]
[2019-08-06T16:16:54,108][INFO ][o.e.p.PluginsService ] [XXXXXXX] loaded module [x-pack-watcher]
[2019-08-06T16:16:54,109][INFO ][o.e.p.PluginsService ] [XXXXXXX] no plugins loaded
[2019-08-06T16:16:58,334][INFO ][o.e.x.s.a.s.FileRolesStore] [XXXXXXX] parsed [0] roles from file [/opt/openv/elk/elasticsearch-7.2.1/config/roles.yml]
[2019-08-06T16:16:59,198][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [XXXXXXX] [controller/19992] [Main.cc@110] controller (64 bit): Version 7.2.1 (Build 4ad685337be7fd) Copyright (c) 2019 Elasticsearch BV
[2019-08-06T16:16:59,620][DEBUG][o.e.a.ActionModule ] [XXXXXXX] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-08-06T16:16:59,789][ERROR][o.e.g.GatewayMetaState ] [XXXXXXX] failed to read or upgrade local state, exiting...
java.io.IOException: failed to find metadata for existing index .kibana_task_manager [location: Yi_UxiceTwS093xLAQJpzw, generation: 30]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:102) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:473) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:251) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.1.jar:7.2.1]
[2019-08-06T16:16:59,797][ERROR][o.e.b.Bootstrap ] [XXXXXXX] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
at org.elasticsearch.node.Node.(Node.java:580) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.1.jar:7.2.1]
"logs/elasticsearch.log" 103L, 12907C

4

Are you sure you deleted everything under /data/nodes/0/? I think you might have left something in /data/nodes/0/_state. You'd be best off deleting the whole 0 folder.

5

You were absolutely right. After deleting the folder it worked.

Thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.