Remove corrupted shard

Elastic Stack Elasticsearch
1

Hi,

we are using ES 8.19.2 on 30 data nodes with ~100TB data. We don’t have any snapshots. Shards per index is 30.

Someone accidentally deleted some indices from filesystem of some of the nodes. (/data/indices/XYZ). Yes, the complete folder. Dont ask me why :slight_smile:

The current situation is: The cluster is in red state, because if the primary and the replica shard was deleted on the random nodes, the data is obviously lost. I’m OK with it.

We still have 29 shards of the 30 shards.
I tried to recover from red state, by using “elasticsearch-shard remove-corrupted-data” to get rid of the now empty shard. But I can not even use it, because it looks like it needs to have the translog file which was also deleted, because the complete folder of the index was deleted.

bin/elasticsearch-shard remove-corrupted-data --index myindex --shard-id 1

Oct 27, 2025 10:33:39 AM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
org.elasticsearch.ElasticsearchException: translog directory \[/data1/indices/7-6OoX2wSs2Ucgv67BFQmA/1/translog\], must exist and be a directory
at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.lambda$processDataPaths$1(RemoveCorruptedShardDataCommand.java:249)
at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.findAndProcessShardPath(RemoveCorruptedShardDataCommand.java:171)
at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.processDataPaths(RemoveCorruptedShardDataCommand.java:245)
at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.processDataPaths(ElasticsearchNodeCommand.java:146)
at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.execute(ElasticsearchNodeCommand.java:164)
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
at org.elasticsearch.cli.Command.main(Command.java:54)
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65)

Is there any other way to “delete” the shard from the index, without losing all the other data from the same index?

Thanks!

1 Like
2

I think I found a solution by using the reroute API and allocating the missing shard as empty shard with "accept_data_loss": true

1 Like
3

Worked out pretty well.

Just for documentation purpose:

POST /_cluster/reroute?metric=none
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "myindex",
        "shard": 0,
        "node": "node1",
        "accept_data_loss": false <-- change this :)
      }
    }
  ]
}
1 Like
4

Yep that’s the right answer. I changed the "accept_data_loss": true line as we’ve had some problems with folks blindly copy-pasting this command in the past.

3 Likes