How to modify ES cluster and repair when unassigned?


(Chenryn) #1

Hi all:
I hava an ES cluster running on two node with ten shards and zero
replica. Onetime I find my page error and then enter es_head and find one
shard was unassigned.
I check es.log but nothing there. At last I found that ES was error
three hours ago by grep my webserver.log
So, is there some scrpt like nagios check_elasticsearch?
And, Is there some way to repair my cluster?

--


(Radu Gheorghe) #2

Hello Chenlin,

On Fri, Oct 19, 2012 at 2:21 PM, chenlin rao rao.chenlin@gmail.com wrote:

Hi all:
I hava an ES cluster running on two node with ten shards and zero
replica. Onetime I find my page error and then enter es_head and find one
shard was unassigned.
I check es.log but nothing there. At last I found that ES was error
three hours ago by grep my webserver.log
So, is there some scrpt like nagios check_elasticsearch?

Actually, there is one called just like that:

Or you can try to use check_http or check_http_json with the Cluster Health API:
http://www.elasticsearch.org/guide/reference/api/admin-cluster-health.html

and grep for "green" in the result, as suggested here:

I also wrote a couple of ES checks here:

They report the total number of documents and the total number of
inserts/sec between checks.

And if you want a more complete monitoring solution for your ES, I'd
suggest you look at SPM for Elasticsearch:

And, Is there some way to repair my cluster?

It's strange that you don't get anything in the logs. I'd try to
increase the logging verbosity (if you installed from the Debian
package that should be in /etc/elasticsearch/logging.yml), then
restart the node with the unassigned shard and see what the logs say
then regarding that shard.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--


(Daniel Maher-3) #3

On 12-10-19 2:21 PM, Radu Gheorghe wrote:

Or you can try to use check_http or check_http_json with the Cluster Health API:
http://www.elasticsearch.org/guide/reference/api/admin-cluster-health.html

I highly recommend check_http_json, mostly because I wrote it for this
exact purpose (monitoring Elasticsearch). :slight_smile: The big selling point is
that it can easily drill down into nested JSON structures and can be
used to monitor totally arbitrary values.

Enjoy !

--
dan (phrawzty).
mozilla webops; european outpost.

--


(system) #4