Multiple indexes break elasticsearch (2.3.1) cluster replication?

Elastic Stack Elasticsearch
1

I have a 3 node Elasticsearch cluster (2 data nodes).

My solution needs to support full replication if any 1 of the 3 nodes goes down or simply shutdown. I've been turning the ES server processes off and on and the replication always works correctly no matter what I do - when I have just 1 index (defaults of 5 primary shards and 1 replica). So everything works great and does what is required.

But a new requirement has just come in to add a second separate index (same defaults of 5 primary shards and 1 replica). But since I added this new index (even though it contains no data) it seems to break the replication on my first index. In that when I test it by shutting down and rebooting nodes I can see data go missing in kibana. Sometimes when I restore the previous master I can see the data come back.

Is this expected behavior in Elasticsearch 2.3.1 - can adding a second index break replication like this?

Thanks

2

Ok I narrowed in further on the issue - it has nothing to do with the second index...

if I reboot the current master - its like I loose the data that was inserted in the last "few" minutes.

Stopping the process and leaving the server running is fine - there is no loss of data,
But shutting down the vm, results in the above loss of data.

3

How many master eligible nodes do you have in the cluster? If this is 2 or 3, have you set discovery.zen.minimum_master_nodes to 2 so you can avoid split brain scenarios?

4

Thanks for your quick reply.
My cluster has 3 eligible master nodes and:
discovery.zen.minimum_master_nodes is set to 2

The issue just seems to be with the cluster node being issued a shutdown command and elasticsearch is not getting to shutdown correctly - if I call a shutdown script when the cluster node is being shutdown - it should fix the issue.

Thanks again

5

Yes doing the following solved the issue:

On each Elasticsearch node, I edited this file:
/usr/lib/systemd/system/elasticsearch.service
And added this line:
ExecStop=/usr/share/elasticsearch/bin/elasticsearch stop

Now when a node is issued a shutdown or reboot command no data is lossed.

Thanks again

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.