I have 2 ElasticCloud on Azure japaneast and southeastasia.
Try to test failover, I want make snapshot on japaneast, then restore on southeastasia.
I know direct restore didn't support between regions, So I make script to copy blob files from japaneast to southeastasia.
After copy files, I feel snapshot is OK on southeastasia.
But after restore snapshot on southeastasia, No snapshot restored status and Index Health is red.
What's wrong with me?
Thanks.
What is the output from the _cat/recovery?v API endpoint?
@warkolm
Thank you for your response.
API result is here:
But red status indices not included.
.security-tokens-7 0 1.1s peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 33 33 100.0% 33 80509 80509 100.0% 80509 0 0 100.0%
.security-tokens-7 0 258ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 33 0 0 100.0% 80509 0 0 100.0%
.kibana-event-log-7.9.3-000001 0 611ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 4 4 100.0% 4 5744 5744 100.0% 5744 0 0 100.0%
.kibana-event-log-7.9.3-000001 0 61ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 4 0 0 100.0% 5744 0 0 100.0%
.security-7 0 1.3s peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 39 39 100.0% 39 198259 198259 100.0% 198259 0 0 100.0%
.security-7 0 299ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 39 0 0 100.0% 198259 0 0 100.0%
.kibana_task_manager_2 0 986ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 27 27 100.0% 27 28047 28047 100.0% 28047 6 6 100.0%
.kibana_task_manager_2 0 97ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 27 0 0 100.0% 28047 0 0 100.0%
.apm-custom-link 0 641ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 1 1 100.0% 1 261 261 100.0% 261 0 0 100.0%
.apm-custom-link 0 27ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 1 0 0 100.0% 261 0 0 100.0%
.kibana_task_manager_1 0 759ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 24 24 100.0% 24 22965 22965 100.0% 22965 0 0 100.0%
.kibana_task_manager_1 0 135ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 24 0 0 100.0% 22965 0 0 100.0%
.apm-agent-configuration 0 613ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 1 1 100.0% 1 261 261 100.0% 261 0 0 100.0%
.apm-agent-configuration 0 36ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 1 0 0 100.0% 261 0 0 100.0%
.kibana-event-log-7.10.1-000001 0 653ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 4 4 100.0% 4 5792 5792 100.0% 5792 0 0 100.0%
.kibana-event-log-7.10.1-000001 0 60ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 4 0 0 100.0% 5792 0 0 100.0%
.kibana_2 0 986ms peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 33 33 100.0% 33 10911883 10911883 100.0% 10911883 0 0 100.0%
.kibana_2 0 301ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 33 0 0 100.0% 10911883 0 0 100.0%
.kibana_1 0 1.1s peer done 10.46.104.24 instance-0000000000 10.46.104.12 instance-0000000001 n/a n/a 30 30 100.0% 30 10921713 10921713 100.0% 10921713 0 0 100.0%
.kibana_1 0 105ms existing_store done n/a n/a 10.46.104.24 instance-0000000000 n/a n/a 0 0 100.0% 30 0 0 100.0% 10921713 0 0 100.0%
What about _cat/allocation?v?
Response is here:
15 21.6mb 44.6mb 59.9gb 60gb 0 10.46.104.24 10.46.104.24 instance-0000000000
15 21.9mb 28mb 59.9gb 60gb 0 10.46.104.12 10.46.104.12 instance-0000000001
4 UNASSIGNED
Just to be sure, I download blobs from japaneast and southeastasia, them compare them.
No difference found.
I try with s3 snapshot repository, I got same result (red state indices).
If your cluster health is red you should always start from the allocation explain API. Here is a blog post about diagnosing red health:
@DavidTurner
Thank you for your reply !
I found some plugins are not installed on southeastasia deployment.
After install plugin, restore is succeeded.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.