Need help for Indexes rotation policy

Elastic Stack Elasticsearch
1

Hello,

I currently use a 3 node ES-cluster for store my Web servers' logs.

3 Master/Data node with same config.

I added a tag to the third node that got more disk than the two other.

Filebeat indexes are created each day, with 1 shards and 1 replicas. (so stored on two nodes).

Currently, I use curator to close indexes (more than 1 week).

My goal :

  • Use the two node (without tag) to store "living" indexes (open)
  • Use the third node (tagged as "big") to store all closed indexes.

I can't figure out how to do that using curator for example.

I tried to use "allocation" to move my shard to the taggued node, it work but my closed index continu to take storage of the node where it was allocated ( I see it in looking on /var/lib/elasticsearch files)...

Any idea how to completly move my closed index to the taggued node without keeping storage on the other ?

Thanks a lot !!

2

Are you closing the index first and then try to move it? It might be slightly surprising, but closed indices are not part of the routing table.

The common problem with that is that if a node containing a copy of a closed index is removed from the cluster, its data will not be replicated to another node. IMO the same problem hits you when you close an index and then try to change the allocation.
The current workaround is probably to change the allocation on day N and only close the index on day N+1. There is also an open issue to change this behavior in the future: https://github.com/elastic/elasticsearch/issues/33888

3

EDIT:

I fixed my issue, here is my current working workflow that Curator do :

  • It move replicas to 0 for +7 days indexes
  • It make the allocation to the taggued node for +7 days indexes
  • It close +7 days indexes

And now it's perfect, single replicas index take hard disk space only on my "big disk" server.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.