CCR# Cross Cluster Replication

Elastic Search
1

Hello,
Have you ever handled the time series indices on the follower cluster post implementing cross-cluster replication?

We have implemented CCR - able to see the indices created on the follower indices.

The issue is - CCR doesn't delete the old indices even though they are curated on Leader cluster.

to delete old indices we need to a) pause the replication b) unfollow the patterns c) delete the old indices.

I wanted to know if there is any one who achieved this via ILM? If yes how?
or
is there a better way to handle this scenario?

Thanks in Advance,
Seshu

2

Hey Seshu,

It looks like you should create the same ILM policy without rollover part in the follower cluster.

3

This document looks good and is helpful for those who are setting up CCR for time series use cases. Thank you.

I have another scenario where full text search usecase-based indices:

eg indices: clientid_indexName_v2-20240807041326

These sorts of indices are created daily via the dotnet service when invoked via scheduler jobs.
Unfortunately,, this scheduled jobs gets killed abruptly if data loads are not finished before business start time, eg; 8AM
This means we can expect to have multiple indices in the cluster with the above-mentioned index pattern.
The good thing is that there is always one good index that has full data and aliases pointed to, and all the old indices are deleted post-full data load completion.

CCR Flow:
CCR auto-follow patterns are able to track these indices and replicate them in a follower cluster.
But old indices are not being cleaned up.

The idea is to set an ILM and clean up indices older than 10 days in the Follower cluster.

The document says it can be done automatically for rollover indices, but unfortunately, our indices are not set for rollover and are created daily.

Can this document be used to set unfollow on the older indices that are older than 10 days and set the delete phase via ILM?

4

Happy to hear the document is useful. From the article, index.lifecycle.indexing_complete and indexing_complete must be true on the leader index to succeed the unfollow.

Can this document be used to set unfollow on the older indices that are older than 10 days and set the delete phase via ILM?

As far as I know, there is no way to set unfollow in the ILM.

Can you check what are these values in leader index settings?

  • If the values are false you should find a way to make them true with a code or with ILM.
5

sorry for getting back to you late.
Indices are set with the following value "indexing_complete": "false".

As you suggested, we will have to see how to update these parameters

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.