In my deployment I have a cluster with two machines with an Elasticsearch installation. I have created one node on every machine so, at the end I have one cluster with two nodes and 10 shards (5 primary and 1 replicas).
I have some doubts related to synchronization after failover. Let me explain:
If one of my elasticsearch machine (e.g Machine1) goes down, I will not have any problem on data recollecting because there is another machine indexing logs (in this case, Machine2). But when Machine1 is up again, there some logs that are contained into Machine2 that are not contained into Machine1.
Is there a way to synchronize data between nodes? I need to store the same information on each node.
Another question regarding nodes,
Can I deploy three nodes in two machines? do you recommend it?
My last question is related to logstash output.
I have configured logstash to send logs to Machine1 (only), and Machine1 replicates data to Machine2...do you suggest to configure logstash to send logs to both machines?
This synchronisation happens automatically when a node reconnects. The node will join the cluster and the master (in your case the other node since it will be the only one that was active) will kick off a recovery which will allocate replicas to the reconnected node and may sure they are up to date with any changes that happened while it had disconnected.
Yes its possible to start two nodes on one machine but it doesn't often make sense since if the machine dies you lose both nodes anyway. Having three nodes across three machines will give you better resiliency if you are able to do that but its not essential. One thing you should make sure is that you have set the minimum_master_nodes setting appropriately for the number of master eligible nodes in your cluster.
Yes, you should set the hosts parameter in the Elasticsearch output in Logstash to contain both nodes so Logstash can continue indexing if one node goes down. AFAIK Logstash we choose one of the hosts and send requests to that node until it is unavailable and then will send requests to the other node when the first disappears
How can I check that information are replicated in both nodes? maybe doing a "du -sh" in "/var/lib/elasticsearch/nodes/0/indices/" to check sizes?
Related to minimum_master_nodes, which could be the best approach? setting minimum_master_nodes to 2 or 1? In my case, I'm seeing that I have defined two nodes with "master, data & ingest" and I dont know if I'm gonna avoid the "split brain".
So the answer on this depends on how much you want to check and the time you are willing to spend checking.
Firstly you can see summary information for all your shards by using the _cat/shards API and you can check the progress of the recovery of shards in the _cat/recovery API, together they will give you a good overview of what ES is doing with your shards at any particular moment.
If you really wanted to check the content was the same between your primaries and replica shard copies you would need to use the scroll API together with the preference option to stream out all the documents for particular shard copies and compare the results to work out if they are the same, but this is not usually necessary.
For a two node cluster you would need to set minimum_master_nodes to 2 to avoid split brain. That would mean that if a single node dies (or becomes partitioned from the other node) you would not be able to elect a master node and therefore would not be able to do any cluster level operations (index level operations such as index requests and search requests would still work so long as they didn't need any resources on the unavailable node.
This is why I mentioned above that it would be more ideal if you could run three nodes on three separate machines as then you would be able to set minimum_master_nodes to 2 and have the resiliency that you could still have cluster level operations working if one node dies or becomes partitioned (since the other two nodes would still be able to form a cluster)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.