Persistence of lifecycle policies accross container rebuild

Elastic Stack Elasticsearch
,
1

Hi,

I have an Elasticsearch server running on a Docker infrastructure with the following containers:

  • Kibana
  • Logstash
  • Elasticsearch

Once my server was set up, I used the Kibana interface to define Lifecycle policies to move my indexes to the Cold phase after 30 days.

Today, I want to update my server, which requires me to recreate the three containers listed above. Of course, my Elasticsearch volume will remain unchanged. Once these containers are recreated:

  • Will my Lifecycle policies still be active?
  • Will my indexes in the Cold phase remain in this phase?

Best regards,

2

How are you starting it? With docker compose?

If you are binding some volume from your host, and this data will still exist, than the policies and indices should persist.

But, are you running everything on the same host? How did you setup the cloud nodes? It is not clear what you are doing in the cold phase.

Normally you move data from hot to warm/cold after sometime and the warm/cold phase are in different nodes with different (slower) hardware, if it is everything in the same host/hardware it does not make much sense to have other data phases.

3

Thanks for your feedback.

Yes, I will use Docker Compose to rebuild and run my containers. Since I configured my Lifecycle Policies through the Kibana interface, I'm not sure whether these policies are stored in my volume. But if they are, they should persist.

Yes, I'm running everything on the same host with 16GB of RAM. Without Lifecycle Policies, my server quickly runs out of memory with the -Xmx8g -Xms8g settings. Since data older than 30 days is not frequently requested, but I still want to keep it for a year, I've set up the following policies:

  • Data older than 30 days is moved to Cold without replicas.
  • Data older than 390 days is deleted.
4

Here is a little feedback.
I have updated my server recreating the container with docker compose.
Everything is fine. As expected, my lifecycles policies are still present and applied, my index remains in the expected phase (cold if older than 30days).