Restore all User created Indices, exclude the indices starting with

[x00m]

I am trying to ALL restore user created indices. I don't want to restore the .geoip_databases, .security, .ds-.logs-deprecation.elasticsearch-default* and so on.

I tried this :-

{
"indices": "*",
"include_global_state": false
}

but I get this error :-

"type": "snapshot_restore_exception",
"reason": "[my_repo:test-snapshot-2/yWuONkjGTIuieFAe7dWeYQ] cannot restore index [.ds-.logs-deprecation.elasticsearch-default-2023.02.01-000001] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name"

So, I tried this :-

{
"indices": "-.ds-.logs-deprecation.elasticsearch-default*, *",
"include_global_state": false
}

But then I get the same error, but stating that an index with name .geoip_databases already exists, and then .security-7

So this is what i tried:

{
"indices": "-.ds-.logs-deprecation.elasticsearch-default*, -.geoip_databases, -.security-7, *",
"include_global_state": false
}

This then returns 404 with error :-

{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index [ -.geoip_databases]",
"index_uuid": "na",
"index": " -.geoip_databases"
}
],
"type": "index_not_found_exception",
"reason": "no such index [ -.geoip_databases]",
"index_uuid": "na",
"index": " -.geoip_databases"
},
"status": 404
}

How to solve this?

Eck 2.3.0, elasticsearch 7.17.8

[jacinto_calvo_sintes]

Hi @x00m,
maybe would be easier to just add the list of the indices you want to restore instead of using * wildcard?
Is the list of indices too large?
you could try something like

{
"indices": [ "logs-*", "index-1", "index-2"] ,
"include_global_state": false
}

Hope this helps.

[x00m]

The list of indices is large (few thousands). I just want to restore all indices except the ones starting with a dot

[jacinto_calvo_sintes]

Hi,
I did some tests and as you said, the _restore option using both wildcards *,-.* doesn't seem to work. At least the tests I did had the same behavior as your case. I found that open issue that might have relation snapshot with exclude expression `-` does not work if you specify explicit index name · Issue #83435 · elastic/elasticsearch · GitHub
I managed to restore the snapshot the way you want doing some steps.

I had to rollover the indices from .ds-logs and ds-ilm to remove the ones that were on the backup, I couldn't exclude for some reason. I then deleted the ds indices from the cluster that were in the backup and run the _restore with the following

POST _snapshot/backup/daily-snap-2023.02.10-y87yklpfra2eh24jar81fa/_restore
{
  "indices": "*,-.*,-.apm*,-.geoip*,-.tasks,-.kibana*,-.ds*",
  "include_global_state": false 
}

Although the ds indices were removed I just kept it in case, I tried with a docker service without authentication, so maybe there are more hiden indices to exclude.
The only ones that gave me problems were the ones starting with ds

[x00m]

@jacinto_calvo_sintes

I had to rollover the indices from .ds-logs and ds-ilm to remove the ones that were on the backup

Can you please shed some more light on what you mean by this?

[jacinto_calvo_sintes]

Hi @x00m , sure.
ds-.logs- indices that your snapshot wants to overwrite are based on ILM templates and they can be rotated.
As in your new cluster you had some of these system indices matching it's name, it couldn't overwrite, so what I did is to rollover the index that matched on the snapshot and the cluster so I could delete from the cluster the one coming from the snapshot.

# check the aliases associated for the rollover
GET .ds-ilm-history-5-2023.02.10-000001 
# rollover the index
POST ilm-history-5/_rollover
# delete old index which is also in snapshots
DELETE .ds-ilm-history-5-2023.02.10-000001 

# same process for .ds indices
GET .ds-.logs-deprecation.elasticsearch-default-2023.02.10-000001

POST .logs-deprecation.elasticsearch-default/_rollover

DELETE .ds-.logs-deprecation.elasticsearch-default-2023.02.10-000001

Now I don't have the same .ds-logs in snaphost and cluster so I can perform the snapshot recovery with

POST _snapshot/backup/daily-snap-2023.02.10-y87yklpfra2eh24jar81fa/_restore
{
  "indices": "*,-.*,-.apm*,-.geoip*,-.tasks,-.kibana*,-.ds*",
  "include_global_state": false 
}

It makes it fine.
These files are logs generated by elastic and some internal work, I don't think they're much important when restoring backups, maybe you could just remove them, but this is a cleaner procedure.
The rest of internal indices shouldn't give any errors.
Hope this helps.

[x00m]

Thank you so much! I'll try this out!