I did a lot of research, and didn't find anything that was 100% clear to me as far as re-deploying a cluster. I found some good tutorials, and blog posts that touch on the subject with insightful information such as disabling shard allocation, stopping an node, and then restarting. I have a static ES cluster; ie nobody or nothing is writing to it at the moment. However, when I re-deploy from scratch it reloads all shards - I'm not sure if these are coming from the s3 repo they were restored from, or if the master is rebalancing.
_cluster/allocation/explain - shows that shards are unassigned.
_cluster/health - shows the primary shard and active shard count increasing as they are distributed across the data nodes.
I have a large number of small indices (not my design) that have 5 shards each and 1 replica, and two data nodes using persistent EBS storage. Here is my current status after a re-deploy:
`{
"cluster_name": "elasticsearch",
"status": "red",
"timed_out": false,
"number_of_nodes": 5,
"number_of_data_nodes": 2,
"active_primary_shards": 5081,
"active_shards": 5081,
"relocating_shards": 0,
"initializing_shards": 8,
"unassigned_shards": 5723,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 8,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 845,
"active_shards_percent_as_number": 46.99408065112838
}`
Side Note - this is data migrated from an AWS ES service cluster into my own ad hoc / managed cluster.
Herein lays my confusion - I thought since I am using persistent data storage that a re-deploy would be almost instantaneous as the shards are already balanced across the data nodes; ie the nodes come up and the shards are already there. This does not seem to be the case. I have tried tweaking my gateway parameters as well as outright disabling shard allocation/rebalance on re-deploy and either it rebalances with the former or does nothing (as expected) with the latter but with no shards whatsoever.
Does anyone know if there is a way to re-deploy an ES cluster, re-attach data volumes, and have the data be auto-discovered by the masters to save load time? Thanks!
Oh, yeah here is my "boot" up elasticsearch.yaml:
`cluster.name: elasticsearch
# shard things:
cluster.routing.allocation.allow_rebalance: "indices_primaries_active"
cluster.routing.allocation.enable: "none"
cluster.routing.rebalance.enable: "none"
node.data: ${NODE_DATA:true}
node.master: ${NODE_MASTER:true}
node.ingest: ${NODE_INGEST:true}
node.name: ${HOSTNAME}
network.host: 0.0.0.0
cloud:
kubernetes:
service: ${SERVICE}
namespace: ${KUBERNETES_NAMESPACE}
# see https://github.com/kubernetes/kubernetes/issues/3595
bootstrap.memory_lock: ${BOOTSTRAP_MEMORY_LOCK:false}
discovery:
zen:
hosts_provider: kubernetes
minimum_master_nodes: ${MINIMUM_MASTER_NODES:2}
readonlyrest:
enable: false
# see https://www.elastic.co/guide/en/x-pack/current/xpack-settings.html
xpack.ml.enabled: false
xpack.monitoring.enabled: ${XPACK_MONITORING_ENABLED:false}
xpack.security.enabled: ${XPACK_SECURITY_ENABLED:false}
xpack.watcher.enabled: ${XPACK_WATCHER_ENABLED:false}
# see https://github.com/elastic/elasticsearch-definitive-guide/pull/679
processors: ${PROCESSORS:}
# avoid split-brain w/ a minimum consensus of two masters plus a data node
gateway.expected_master_nodes: ${EXPECTED_MASTER_NODES:3}
gateway.expected_data_nodes: ${EXPECTED_DATA_NODES:2}
gateway.recover_after_time: ${RECOVER_AFTER_TIME:3m}
gateway.recover_after_master_nodes: ${RECOVER_AFTER_MASTER_NODES:1}
gateway.recover_after_data_nodes: ${RECOVER_AFTER_DATA_NODES:1}
`
I've tried with and without the:
` cluster.routing.allocation.allow_rebalance: "indices_primaries_active"
cluster.routing.allocation.enable: "none"
cluster.routing.rebalance.enable: "none"`
Yet either case it seems to rebalance, or require a rebal.