ES cluster in Amazon auto scaling group


(Bastien Chong) #1

Hi,

I have setup an autoscaling group in Amazon for my Elasticsearch Cluster.
Everything was working fine with the discovery and all.

I currently have 2 instances in the AutoScaling group. I started to work on
a new Elastic Load Balancer health check, but since it's not done yet, the
check only pass for 1 node and the other is terminated every 3000 seconds.

My cluster is configured with the default settings, 5 shards, 1 replica. So
in theory, when 1 node is terminated, the fist one should keep all the data
and automatically know that it's alone in the cluster. When a new node is
spawned, it will automatically add it and sync.

During the WE, this continued to happen, but I'm now in a situation where
my cluster is in a bad state, my indexes doesn't have 5 shards but 3,
Elastic HQ plugin tells me there's more than 2 nodes, unassigned shards.

This test cluster doesn't have any data, less than 1000 documents, so the
replication time is minimal.

What do you recommend ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/442caee3-4c6c-4ff8-b672-593e851f861f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Bastien Chong) #2

Is it possible that when 2 nodes were up, somehow some shards an their
replica ended up on the same node, so when it was terminated, I lost data ?

Isn't ES supposed to balance shard/replica in a way that this won't happen ?

On Monday, July 14, 2014 9:52:12 AM UTC-4, Bastien Chong wrote:

Hi,

I have setup an autoscaling group in Amazon for my Elasticsearch Cluster.
Everything was working fine with the discovery and all.

I currently have 2 instances in the AutoScaling group. I started to work
on a new Elastic Load Balancer health check, but since it's not done yet,
the check only pass for 1 node and the other is terminated every 3000
seconds.

My cluster is configured with the default settings, 5 shards, 1 replica.
So in theory, when 1 node is terminated, the fist one should keep all the
data and automatically know that it's alone in the cluster. When a new node
is spawned, it will automatically add it and sync.

During the WE, this continued to happen, but I'm now in a situation where
my cluster is in a bad state, my indexes doesn't have 5 shards but 3,
Elastic HQ plugin tells me there's more than 2 nodes, unassigned shards.

This test cluster doesn't have any data, less than 1000 documents, so the
replication time is minimal.

What do you recommend ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/77c2be3c-0013-43d1-a513-6778d4682cfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3