Is there any way to keep indices older than some days in one cluster and other indices in another cluster?

Elastic Stack Elasticsearch
1

Hi,

I have an ES architecture like below;

arch

I have named my indices based on day, like myindexa-2017-11-08, myindexa-2017-11-09, myindexb-2017-11-08, myindexb-2017-11-09 etc. I would like to keep indices older than 10 days in C2 and other indices in C1. Is there any way or workaround to achieve this?

Thanks.

2

That would require you to use snapshot and restore to move indices between clusters. A more common approach is to have data of different ages reside on different set of nodes in the same cluster, which is what we often refer to as a hot-warm architecture.

3

Hi @Christian_Dahlqvist , thanks for your reply.

Currently I am using this snapshot - restore technique. I take snapshot of indices of 1 cluster C1 for a particular day, daily, and zip that folder. Then I copy that folder to C2, extract the zip, and restore indices. I am doing this because the space in C1 is limited.

The issues I faced here was;

  1. I need to use 2 Kibana's to access indices in these VMs
  2. I need to run some cron jobs to take snapshots, make zip, delete folder, upload to sftp, download from sftp, extract, perform restore, delete folder. I would like to minimize the steps.

These are some reasons why I asked this question. Is there any way to go this "smoothly", like data sync b/w ES nodes?

4

Can somebody kindly help me on this?

5

Why are you using 2 clusters instead of one larger one? Are they located far apart? Have you looked into querying both clusters using cross-cluster search?

As far vas I know there is no way to seamlessly move indices between clusters apart from snapshot and restore. This is not instant nor automatic, but you could streamline your process by e.g. using S3 for the snapshots to avoid having to copy data manually.

1 Like
6

Thank you @Christian_Dahlqvist , I understand. The clusters are in the same data center, but there are restrictions because of some security parameters. Need to look into cross-cluster search.

7

Cross-cluster search will help you query the data across both clusters at the same time, but will not help you move indices.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.