Snapshot indices to S3 repository_exception

[rymr]

Hi,

I backup my indices with a script on s3 bucket following those steps:

• create repository if it doesn't exist
• cleanup the repository
• remove old snapshots
• snapshot in 4 repositories depending of indices types

The first run works well but on the second one, I have this error:

Failed to create snapshot.Status: 500 - {
   "error":{
      "root_cause":[
         {
            "type":"repository_exception",
            "reason":"[<REPOSITORY_1>] Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repository by a process other than this cluster or an issue with the repository's underlyingstorage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents."
         }
      ],
      "type":"repository_exception",
      "reason":"[<REPOSITORY_1>] Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repository by a process other than this cluster or an issue with the repository's underlyingstorage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents."
   },
   "status":500
}

When I remove it from the cluster DELETE _snapshot/<REPOSITORY_1>/ and relaunch my script, the workaround works well.

I noticed something in my snapshots. The repositories structure was built to be like :

REPOSITORY_1/
├── snap-repository_1.1
└── snap-repository_1.2
REPOSITORY_2/
├── snap-repository_2.1
└── snap-repository_2.2
...

But, in the reality, each snapshot is in each repository

REPOSITORY_1/
├── snap-repository_1.1
├── snap-repository_1.2
├── snap-repository_2.1
└── snap-repository_2.2
REPOSITORY_2/
├── snap-repository_2.1
├── snap-repository_2.2
├── snap-repository_1.1
└── snap-repository_1.2
...

So my guess is, since when I remove old snapshots, it remove them from one repository but not from another. So it seems logical that ES disable it.

So I have two questions:

• snapshots aren't supposed to be stored only in one repository as asked in the request?
• is it the root cause for the repository_exception ?

Elasticsearch version: 7.9

Thanks.
Rym R.