Not include all indices in ECE snapshot repo S3

[markn (Mark Niemeijer)]

Hi,

back again. Now got the AWS S3 repo working behind corporate proxy. Snapshots are running. By default it snapshots all indices, we could not find a way to only backup the system indices.

We searched the docs and this forum, no luck yet.

We have a use case where we are not allowed to backup the data indices to AWS (and there is also no need for).

Thanks.

Cheers,
Mark

[Alex_Piggott]

Glad you got the S3 repo working (thanks for posting the resolution in the other thread)

Currently ECE does not support filtering on indices for the system managed snapshot repo

There are a couple of options:

• Do not use the ECE snapshot mechanism, just use SLM with an index filter (this has the downside of requiring that restore actions be performed manually vs via ECE)
• I believe the Kibana SLM UI blocks setting an index filter, but the ES SLM API does not.

[markn (Mark Niemeijer)]

Hi Alex,

thanks! We did some tests this morning. And we are able to connect to S3 AWS via corporate proxy if we just create the S3 repository via Kibana Console.

 PUT /_snapshot/aws-s3-direct
 {
   "type": "s3",
   "settings": {
     "bucket": "ece-snapshot-repo",
     "region": "region",
     "access_key": "xyz",
     "secret_key": "abc",
     "proxy.host": "10.10.a.b",
     "proxy.port": "8080"
   }
 }

From there we can create the Snapshot Policies and both work in parallel, the ECE snapshot policy includes all indices. And the "Kibana Console" repo / policy only the system indices. See screenshot.

This means we can use ECE snapshot for customers that do not have an issue that their data is snapshot to S3. For users that don't care about data backup we can use the ES API S3 repo for system indices snapshot only. And for customers that need local DC backup, we can use ES API (and yaml) config for a local filesystem backup solution.

Kind regards,
Mark

P.S. via the Kibana GUI we can select indices for the repo created via ES API.