I am now setting up archival strategy here to take snapshot of all indices including the cluster state, but i am getting frequent shard allocation errors.
The default no of shards allocated is 5 and i have reduced the no of replica's to 1.
Also after setting up the backup repository, i did take few backup's for a cluster of indices but at on point i am getting a message "concurrent_snapshot_execution_exception" which i killing me in proceeding further. I triedto query the status,but still couldn't get any information except snapshot
Is there anything which i could do here which would help me to proceed further and resolve all these problems, I am planning to reduce the no of shards to 3(will i be able to achieve it) and i will also setup cronjobs and curator soon here.
Restores are, by default, asynchronous operations: "success" means the cluster has successfully started to restore the snapshot, not that it has completed. If you want to wait for the restore to complete, set the wait_for_completion parameter:
POST /_snapshot/kindex_backups/snapshot_jan19_index-name/_restore?wait_for_completion=true
{
"indices": "index-name-2019.01.11"
}
Additionally you can monitor the progress of the restore operation using the indices recovery API, or by waiting for all the shards to be allocated using the cluster health API:
Hi Turner,
Thanks for the information, I understand that the snapshot can only be restored when the cluster status is green. But i have a problem here, my cluster status is showing up Red for a long time and snapshots are little baby steps which i have taken to remove old and unwanted indices so i can save some space in the allocated servers. Also i see this unassigned shards coming up more frequently which i believe is the cause for the cluster being Red.
You have over 10,000 active shards in this 3-node cluster. This is far too many and will cause you problems. Here is an article that gives more detail:
I don't think this is the case. It should be possible to restore a snapshot whatever your cluster health (as long as it has a master, of course).
Indeed, a health of red means there are unassigned primary shards. To find the shards that are unassigned you can use GET _cat/shards, and then you can investigate why they are unassigned using the allocation explain API.
Hi Turner,
Thanks for pointing the problem of accumulating indexes within 3 node production cluster, i am tryning to look into the possible resolution steps for the same.
I am planning to reduce the no of shards to be 3 for all indexes and will only one replica.
But my current status of the cluster seems to red because of these many shards, Considering this what would be the best option to take this forward
shrink indexes or reindexing!!
Also,since i have snapshot backup of all january indices can i go ahead and delete them to reduce the no of shards!! -- I am asking this beacuse i haven't tested the restore activity yet.
And finally can i move the contents of snapshot repository to another backup server completely.
Hi David,
After long struggle, i can only reach status where my cluster status has turned yellow and still there too many open shards available, I will consider your valuable suggestion of changing all daily indices into weekly ones, while i start doing it
how can i change their defaults shard allocation to be 1 along with replica as 1!!
It's really hard to help here because you're not sharing much information and are possibly jumping to conclusions. A yellow health means there are unassigned shards, but this isn't normally because there's too many open shards. As I said above:
The number of shards comes from the index settings which probably come from the index template. To adjust the default number of shards, change the index template.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.