Snapshot restoration failures - FileAlreadyExistsException

[Vamsi_Htc]

We are facing issues with restoring a snapshot from s3 bucket and the we are seeing below error.

ES version 5.6.3

Two of our nodes went down which got both primary and replicas which resulted in cluster turning into red . We opted for snapshot restore but failed due to below . Do we always need to delete index before performing snapshot restore .

On some occasions we tried to restore snapshot on existing index with a week old snapshot and worked without any issues .

[2021-07-11T23:12:09,172][WARN ][o.e.c.a.s.ShardStateAction] [DvtNg_b] [default-index][0] received shard failed for shard id [[default-index][0]], allocation id [PUCjSwFbS7Oj2ELhKuN4oA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[default-index][0]: Recovery failed on {QJIcKkY}{QJIcKkYiSd-WgZAVbCJNpw}{dWA1m7CTROyVUybD4wgtqw}{10.XX.XXX.XXX}{10.XX.XXX.XXX:9300}{aws_availability_zone=eu-west-1c}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot_1626033808/dkGnKrdRSb-62t-cO5zcoA]]; nested: IndexShardRestoreFailedException[Failed to recover index]; nested: FileAlreadyExistsException[/var/lib/elasticsearch/nodes/0/indices/NE-108vTRhaZvja7btbylg/0/index/_gt34t.cfs]; ]

[DavidTurner]

Not sure what version you're really using (5.7.3 was never released) but the last version that ever emitted this message is 2 years past EOL. You should upgrade. I can't say much about such ancient versions, but can confirm that in recent versions there's no need to delete an index before restoring it from a snapshot.

[Vamsi_Htc]

COrrection the verison is 5.6.3 and there are some plugin dependencies which is stopping us for upgrade . we have plans to upgrade this in future . Can u shed some light on what is best approach to avoid such errors .

[DavidTurner]

I have no better advice than to upgrade, sorry.