Hello,
I'm experiencing an error where my elasticsearch (6.7) data nodes won't start due to name duplication.
Caused by: java.lang.IllegalStateException: index and alias names need to be unique, but the following duplicates were found [mosquitto (alias of [mosquitto-2019.06.20-000005/YprlN_CSRxW7cKQcvpemCw]),mosquitto (alias of [mosquitto-2019.07.04-000007/P0LyDwz-RfeyM7IV2cWmMA]),logback-staging (alias of [logback-staging-2019.05.02-000001/duIUpr5fQmW809hj1ipMIQ]),mongo (alias of [mongo-2019.07.02-000007/-yXUiUnzQjuR3vm3OV_ENw]),mongo (alias of [mongo-2019.06.11-000004/OSbuKOnDQWOObszcx5M7Yg]),mosquitto (alias of [mosquitto-2019.07.11-000008/Q4MLQyZrTmGnvB0Meqy0mw]),mongo (alias of [mongo-2019.06.25-000006/l1uFrlV4S22WeSogqyeh8g]),logback-staging (alias of [logback-staging-2019.06.01-000002/WJhf4V8ZS4-n-SFXwI7taA]),logback-staging (alias of [logback-staging-2019.07.01-000003/9m9qlNU1TI2oC4ImpIkiCA]),mongo (alias of [mongo-2019.07.09-000008/KLXeqcBGSMaQvGdZ08k-iQ]),mongo (alias of [mongo-2019.06.18-000005/nTpLv7EkTsSMd_haDq3MDw]),mosquitto (alias of [mosquitto-2019.06.13-000004/XQfsJ321QSm7c-VobVoEVg]),mosquitto (alias of [mosquitto-2019.06.27-000006/4skwEZTXQhuobJZM4kalwQ])]
at org.elasticsearch.cluster.metadata.MetaData$Builder.build(MetaData.java:1118) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:73) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.1.jar:6.7.1]
... 6 more
[2020-07-09T17:05:19,337][INFO ][o.e.x.m.p.NativeController] [es-data-1] Native controller process has stopped - no new native processes can be started
In the meantime my master is continuously reporting such errors:
[2020-07-09T17:16:56,129][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [es-master-74bf46467d-j7r5l] collector [cluster_stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.1.jar:6.7.1]
(...)
[2020-07-09T17:17:06,131][ERROR][o.e.x.m.c.c.StatsCollector] [es-master-74bf46467d-j7r5l] collector [ccr_stats] timed out when collecting data
[2020-07-09T17:17:06,132][ERROR][o.e.x.m.c.m.JobStatsCollector] [es-master-74bf46467d-j7r5l] collector [job_stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:191) ~[elasticsearch-6.7.1.jar:6.7.1]
(...)
[2020-07-09T17:17:16,116][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [es-master-74bf46467d-j7r5l] collector [index_recovery] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:191) ~[elasticsearch-6.7.1.jar:6.7.1]
(...)
[2020-07-09T17:17:16,117][ERROR][o.e.x.m.c.i.IndexStatsCollector] [es-master-74bf46467d-j7r5l] collector [index-stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:191) ~[elasticsearch-6.7.1.jar:6.7.1]
(...)
I have a 3-master + 2-data + 1-client cluster. This error happens on both of my data nodes.
Is there any way to recover from this without data loss?