# Dec 15th, 2019: \[EN\] Elasticsearch Snapshot Lifecycle Management (SLM) with Minio.io S3

**URL:** <https://discuss.elastic.co/t/dec-15th-2019-en-elasticsearch-snapshot-lifecycle-management-slm-with-minio-io-s3/210608>\
**Category:** Advent Calendar\
**Created:** [December 15, 2019, 8:00am UTC](https://discuss.elastic.co/t/dec-15th-2019-en-elasticsearch-snapshot-lifecycle-management-slm-with-minio-io-s3/210608 "2019-12-15T08:00:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [December 15, 2019, 8:00am UTC](https://discuss.elastic.co/t/dec-15th-2019-en-elasticsearch-snapshot-lifecycle-management-slm-with-minio-io-s3/210608/1 "2019-12-15T08:00:03Z")

</div>

Snapshot Lifecycle Management (SLM) is a way to schedule and define snapshot policies using the Elasticsearch API or the Kibana UI. It also allows to check the existing snapshots and restore them directly on Kibana.

SLM requires a [_basic_](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/start-basic.html#start-basic) license. [Snapshot/Restore](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html) functionality do not require any license and it's available since Elasticsearch 1.4.

SLM has been introduced on the [7.4.0 release](https://www.elastic.co/blog/elasticsearch-7-4-0-released) and further improved on 7.5.0 (with the introduction of a [retention policy](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/slm-retention.html)).

This article will explain how to setup a S3 repository on Elasticsearch targeting a single node [Minio.io](http://Minio.io) instance and define a snapshot policy with SLM.

As a prerequisite, you must be running Elasticsearch and Kibana 7.4 or 7.5 with at least _basic_ license.

Minio will be running on [Docker](https://docs.docker.com/v17.09/engine/installation/). It is not mandatory to use Docker but this guide covers only this kind of setup.

As a reminder, please consider the S3 Repository plugin is tested with AWS S3.

# Installing the S3 Repository plugin

In order to target an S3 repository, we need to install the [S3 Repository plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3.html) on **all** the nodes of our Elasticsearch cluster.  
Each Elasticsearch node **must be restarted** after the installation of the plugin.

The binary `elasticsearch-plugin` is located in the installation path of Elasticsearch, which depends on the [installation](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) method you've chosen.

1. Run the following command to install the plugin.

```bash
sudo bin/elasticsearch-plugin install repository-s3

```

1. Restart the Elasticsearch instance.

2. Repeat for all the nodes.

3. Verify the plugin is installed on all the nodes using the API `GET _cat/plugins?v`.

Notes:

- If you're in an air-gapped environment, please check the offline installation in the documentation.
- If you're using Docker, you can create a derived image using:

```auto
FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.0
RUN bin/elasticsearch-plugin install -s -b repository-s3

```

- Remember you'll need to upgrade the plugin to keep it aligned with your Elasticsearch version as detailed in the [rolling upgrades documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html).

# Installing a single Minio node on Docker

For the scope of this tutorial, we'll be running a single Minio node using Docker.

This is not a production-ready setup and please follow the official [Minio documentation](https://docs.min.io/docs/minio-server-configuration-guide) for a safe and reliable setup.

1. Start our local Minio instance, you can use:

```auto
docker run -p 9000:9000 -e "MINIO_ACCESS_KEY=testkey" \
 -e "MINIO_SECRET_KEY=testsecret" -v /mnt/minio/data:/data \
minio/minio server /data

```

1. Verify it is correctly working accessing `http://<ip of the minio instance>:9000` on your browser

2. Create a bucket clicking on the `(+)` icon on the bottom right of the web UI and selecting `Create bucket`. Type `testbucket` and press `enter`.

3. A new bucket named `testbucket` will appear on the Minio web UI.

# Configure the S3 Client and Repository on Elasticsearch

The S3 Repository plugin is designed to work with the AWS S3 service and it's setup is documented [here](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-usage.html).

In order to make it work our local Minio instance, we need to define a custom S3 client, which we will call `myminio`.

1. Add the following line to `elasticsearch.yml` on every node of the cluster

```auto
s3.client.myminio.endpoint: http://<ip of the minio instance>:9000
# the following line is required since 7.5.0, see https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.4.html#_the_s3_repository_plugin_uses_the_dns_style_access_pattern_by_default
s3.client.myminio.path_style_access: true 

```

1. Add the credentials to the [secure keystore](https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html). Execute those commands with the same user which is running Elasticsearch (`elasticsearch` by default).

```bash
bin/elasticsearch-keystore add s3.client.myminio.access_key
# testkey
bin/elasticsearch-keystore add s3.client.myminio.secret_key
# testsecret

```

1. Restart every node of the cluster. It is necessary for `elasticsearch.yml` changes, as the keystore settings we're using are [reloadable](https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html#reloadable-secure-settings).

Notes:

- Minio supports DNS style access but requires you to setup a FQDN associated to the IP where Minio is running. This is out of scope for this tutorial.

# Create a repository using the SLM UI in Kibana

Execute those steps as `elastic` user (which has `superuser` role) or check the notes below.

1. Access Kibana under `Management / Snapshot and Restore / Repositories` tab
2. Select `Register a repository`
3. As `Repository name` provide an arbitrary name (e.g. `minio`)
4. Select `Repository type` select `AWS S3`
5. Click `Next`
6. Input the `Bucket` name, which in our case is `testbucket`
7. Input the `Client` name, which in our case is `myminio`
8. Click `Register`
9. Click `Verify repository` to ensure the S3 repository is ready to be used

# Create a Snapshot policy

Execute those steps as `elastic` user (which has `superuser` role) or check the notes below.

1. Access Kibana under `Management / Snapshot and Restore / Policies` tab
2. Select `Create a policy`
3. Input the `Policy name`, in our case `test-snapshot`
4. The `Snapshot name` will be the name of the generated snapshot. It supports the [date math](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html) and for this test we can use `<daily-snapshot-{now/d}>` (it will include the date in its name)
5. As schedule, you can set a cron pattern or choose every `hour` for example.
6. Click `Next`
7. On the `Snapshot settings` tab, keep everything as default or customize the index patterns to be included in the snapshot.
8. For this test, deselect the `All indices, including system indices`  
 ![unnamed](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fb0809e4dd197a4864dd46be452242c76a81f2fc.png)
9. Select `Use index patterns`
10. Input `.kibana*`: we will snapshot all the dashboards and visualizations
11. Click `Next`
12. On the `Snapshot retention` tab, set `Expiration` of `2 hours` and set `Snapshots to retain` to `1` (`Minimum count`) and `2` (`Maximum count`). It means: keep all the snapshots which are 2 hours old, always keep at least 1 and at maximum 2 successful snapshots, even if they're older than 2 hours.
13. Click `Next`
14. Check the recap and click `Create policy`
15. Wait until the `Next snapshot` or trigger it manually using the `Run now` icon in the `Policy` tab (it is a ▶ symbol on right side of the new policy you've created).
16. A snapshot will appear on the `Management / Snapshot and Restore / Snapshots` tab and you'll be able to check all the details

Notes:

- Please be aware the `Retention` policy is enforced once per day by default (see `Policies` tab) for all the policies. You can trigger it manually via the SLM UI or change its scheduling. You can trigger the snapshot retention policy manually also issuing the request `POST _slm/_execute_retention` ([documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/slm-api-execute-retention.html))
- Snapshots within the same repository are [incremental](https://www.elastic.co/guide/en/elasticsearch/reference/1.3/modules-snapshots.html#_snapshot)
- Do not manually edit or delete the files saved in S3
- All the operations done via the SLM UI can be done via the SLM APIs ([documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/snapshot-lifecycle-management-api.html)). You can check the policy we've created using `GET _slm/policy/test-snapshot`
- Please follow the instructions [on the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/getting-started-snapshot-lifecycle-management.html#slm-and-security) on how to setup a dedicated user with the rights to perform snapshots and manage SLM policies, repositories and snapshots.
- When using Minio S3 with TLS and you are not using publicly trusted certificates, you'll need to add the public CA certificate to the JVM truststore used by Elasticsearch. This is out of scope for this tutorial.

# Recap

This tutorial should get you up and running with SLM.  
Check out the other supported repositories in our [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#snapshots-repositories):

- `fs` (shared file system), which requires every Elasticsearch node to mount the same shared disk (SMB, NFS)
- `gcs`, Google Cloud Storage repositories
- `hdfs`, Hadoop environments
- `azure`, Azure storage repositories

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:36am UTC](https://discuss.elastic.co/t/dec-15th-2019-en-elasticsearch-snapshot-lifecycle-management-slm-with-minio-io-s3/210608/2 "2022-11-04T07:36:07Z")

</div>