We have a cluster of 2 ElasticSearch nodes operating on production and
handling data for various public transport information systems. It's
working for about a year now, but from time to time one of the nodes turns
yellow and is missing data, which causes serious issues for the clients and
passangers (the requests that go to the yellow node are getting no results
in response). In such case we restart the nodes and reindex which fixes the
problem. But still, this is happening by average once per 2 weeks, so I
wanted to ask for some help.
There also seems to be a problem with the logger, as for some days the log
files are missing, which makes it more difficult to diagnose issues.
What we already found is that the nodes sometimes seem to be so busy, that
pings between them are getting timeouts. Also from time to time one of the
nodes removes the other from it's view (3 ping timeouts in a row), but
usually they join shortly. However sometimes the nodes can't join, one of
them has state yellow, and is
With only 2 nodes, you have no way to defend against a split brain. Using
the minimum_master_nodes property, and a minimum of 3-nodes (odd-numbered
required) you can defend against this case:
though I still personally believe there is a problem with Zen and it's
master election, see Issue #2117 and #2488, you're at least likely to be
covered better than you are using minimum_master_nodes.
We have a cluster of 2 Elasticsearch nodes operating on production and
handling data for various public transport information systems. It's
working for about a year now, but from time to time one of the nodes turns
yellow and is missing data, which causes serious issues for the clients and
passangers (the requests that go to the yellow node are getting no results
in response). In such case we restart the nodes and reindex which fixes the
problem. But still, this is happening by average once per 2 weeks, so I
wanted to ask for some help.
There also seems to be a problem with the logger, as for some days the log
files are missing, which makes it more difficult to diagnose issues.
What we already found is that the nodes sometimes seem to be so busy, that
pings between them are getting timeouts. Also from time to time one of the
nodes removes the other from it's view (3 ping timeouts in a row), but
usually they join shortly. However sometimes the nodes can't join, one of
them has state yellow, and is
Are you monitoring CPU and heap on the nodes? What happens to CPU and java
heap just before nodes turn yellow?
On Monday, April 15, 2013 11:35:10 PM UTC-4, tallpsmith wrote:
With only 2 nodes, you have no way to defend against a split brain. Using
the minimum_master_nodes property, and a minimum of 3-nodes (odd-numbered
required) you can defend against this case:
though I still personally believe there is a problem with Zen and it's
master election, see Issue #2117 and #2488, you're at least likely to be
covered better than you are using minimum_master_nodes.
We have a cluster of 2 Elasticsearch nodes operating on production and
handling data for various public transport information systems. It's
working for about a year now, but from time to time one of the nodes turns
yellow and is missing data, which causes serious issues for the clients and
passangers (the requests that go to the yellow node are getting no results
in response). In such case we restart the nodes and reindex which fixes the
problem. But still, this is happening by average once per 2 weeks, so I
wanted to ask for some help.
There also seems to be a problem with the logger, as for some days the
log files are missing, which makes it more difficult to diagnose issues.
What we already found is that the nodes sometimes seem to be so busy,
that pings between them are getting timeouts. Also from time to time one of
the nodes removes the other from it's view (3 ping timeouts in a row), but
usually they join shortly. However sometimes the nodes can't join, one of
them has state yellow, and is
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.