Snapshot, Hot/Warm Architecture and Upgrade

[yuswanul]

Hi there,

I have a few questions here:

• First, if I have an index of 4.5 TB, what is the best way to back up that much data?
• Second, my existing cluster has 25 data nodes in total, if I want to apply hot/warm architecture, what is the best practice for it?
• Third, currently, my cluster is running on version 7.17. if I want to upgrade my cluster which contains 25 data nodes, 3 master nodes, 2 coordinate nodes, and 1 monitoring site. what is the best practice to upgrade it?

your answer will mean a lot
Thanks

[Christian_Dahlqvist]

@yuswanul

First, if I have an index of 4.5 TB, what is the best way to back up that much data?

The only supported way to backup Elasticsearch is through the snapshot API.

Is this a single index or a set of time-based indices? How many primary and replica shards do you have?

@yuswanul

Second, my existing cluster has 25 data nodes in total, if I want to apply hot/warm architecture, what is the best practice for it?

A hot warm architecture generally assumes you have time-based indices of some sort and a defined retention period. Is that the case for your use case?

[yuswanul]

@Christian_Dahlqvist

The only supported way to backup Elasticsearch is through the snapshot API.

I want to know:

• What is the performance of backup & restore based on, and
• What are the parameters to speed up the process?

@Christian_Dahlqvist

Is this a single index or a set of time-based indices? How many primary and replica shards do you have?

that is a single index and it has 25 primary and replicas. the store.size is 4.5 TB and pri.store.size is 2.2 TB

@Christian_Dahlqvist

A hot warm architecture generally assumes you have time-based indices of some sort and a defined retention period. Is that the case for your use case?

umm no, what I want to know is, with my current cluster condition, which is all data nodes in the hot tier by default (CMIIW). because I don't explicitly set the tier for each data node, how do I make the transition to implement this hot/warm architecture?

[Christian_Dahlqvist]

@yuswanul

I want to know:

• What is the performance of backup & restore based on, and
• What are the parameters to speed up the process?

It depends on a lot of factors, e.g. infrastructure and available resources, so I would recommend you set up a repository and test.

@yuswanul

that is a single index and it has 25 primary and replicas. the store.size is 4.5 TB and pri.store.size is 2.2 TB

Sounds like you have very large shards, which can cause performance problems.

@yuswanul

umm no, what I want to know is, with my current cluster condition, which is all data nodes in the hot tier by default (CMIIW). because I don't explicitly set the tier for each data node, how do I make the transition to implement this hot/warm architecture?

If you have a large single index a hot-warm architecture does not make any sense. What are you hoping to achieve? What is the problem you are looking to solve?

[yuswanul]

@Christian_Dahlqvist

Sounds like you have very large shards, which can cause performance problems.

yeah, pretty large. each shard has a size of 93-94 GB

@Christian_Dahlqvist

If you have a large single index a hot-warm architecture does not make any sense. What are you hoping to achieve? What is the problem you are looking to solve?

actually, it's not the only one. there are a few indexes with a size of 4.2 - 4.5 TB and what I want to achieve is to extend the retention of it

[yuswanul]

regarding extending retention, do you have any other suggestions for that?

currently, all those big indexes have a retention of 15 days

[Christian_Dahlqvist]

It sounds like you are not using time-based indices, is that correct? If so, do you delete data through delete-by-query?

Is your data immutable or are you performing updates?

[yuswanul]

@Christian_Dahlqvist

It sounds like you are not using time-based indices, is that correct?

no, we have been using time-based indices from the start till now. that big data is really the data that comes in 1 day. here is the capture

[Christian_Dahlqvist]

I can see a few deleted documents in the indices. Does this mean that you are performing updates/deletes or may this be a side effect of you specifying your own document ID?

[yuswanul]

yeah, but what is the correlation between deleted documents and backup all those indexes?

[Christian_Dahlqvist]

This is not related to backup, but switching to a hot/warm architecture.

For backup speed you will need to test.

[yuswanul]

@Christian_Dahlqvist

a side effect of you specifying your own document ID

OK, I can confirm that it is a side effect of specifying my own document ID