Moving data from hot to warm node

Elastic Stack Elasticsearch
1

Hi,

We are using curator job to move data from hot to warm nodes.
As of now daily indices are getting created,Since the shard count is high we are planning to change the indexing strategy from daily index creation to monthly.

I would like to know Since the indices would become read only only after the end of the month,Can we move the active(read-write) index from hot to warm node to reduce space consumption on hot nodes?

If yes,will it affect the performance of the cluster.

All these while since indices were getting created daily we used to move 4 days older indices from hot to warm.

2

If the hot nodes can not hold all the data if switching to monthly indices that kind of defeats the purpose. I would generally not recommend moving indices still being written to to warm nodes as these typically have slower storage.

You could instead switch to using the rollover API to get indices of a certain target size. This means each index could cover a variable time period, but Curator contains support for basing action based on the age of the index or the date interval contained in the index, so you should be able to configure it to move indices e.g. once it contains no data newer than X number of days.

3

Thanks for the update,We have plans to implement rollover but it will take time.

As you mentioned above, Moving active indices from hot to warm will affect the I/O operation due to slower storage.

I would request you to confirm if data ingestion will be affected while active index is being moved from hot to warm.

We are planning to move active index only if there is any space scarcity on hot nodes.

4

Anything that uses resources, indexing into warm node or moving shards between nodes, can affect performance. If implementing rollover will take a while, maybe switching to weekly indices and perhaps reducing the number of primary shards is a good intermediate step?

5

Thanks for update, Switching to weekly index looks like good intermediate step.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.