Recovering elasticsearch cluster

Elastic Stack Elasticsearch
1

I am getting below exception for my elasticsearch cluster. Due to this exceptions it is blocking all the incoming traffic and logs are not seeing in Kibana.

[2015-11-24 15:36:34,132][INFO ][cluster.metadata ] [node1.domain.com] [logstash-2015.11.23] creating index, cause [auto(bulk api)], templates [logstash], shards [8]/[1], mappings [default, httpd_search]
[2015-11-24 15:36:35,053][DEBUG][action.admin.indices.stats] [node1.domain.com] [indices:monitor/stats] failed to execute operation for shard [[logstash-2015.11.23][5], node[stz9f0JdTp2YNstANI83WQ], [R], v[3], s[INITIALIZING], a[id=lZ7ECXZ0SiyNT44yAfv-jQ], unassigned_info[[reason=INDEX_CREATED], at[2015-11-24T21:36:34.133Z]]]
[logstash-2015.11.23][[logstash-2015.11.23][5]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:399)
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:376)
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:365)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [logstash-2015.11.23][[logstash-2015.11.23][5]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]

curl -XGET http://node1.domain.com:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch-npe",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 19,
"active_shards" : 38,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

gateway.expected_nodes: 1
gateway.expected_master_nodes: 1
gateway.recover_after_nodes: 1
gateway.recover_after_master_nodes: 1
gateway.recover_after_data_nodes: 1
discovery.zen.ping.unicast.hosts: ["node1.domain.com", "node2.domain.com"]
node.max_local_storage_nodes: 1

Raghu

2

What version?

3

I am using 2.0 version

4

Is this issue because of posted here Marvel license issue

5

Even after upgrading ElasticSearch and Logstash to verions 2.1 and Kibana to 4.3, i am still getting these exceptions.

In logstash i could see below error message.

"retrying failed action with response code: 503".

Even after following steps issue is not getting resolved and log events are not getting indexed.

  1. I have disabled the marvel agent.
  2. Deleted .kibana index to create new indexes.
  3. restarted all the services.

Please help me out in resolving this issue.

6

What's the output from _cat/shards look like?

7

All the Indexes shards status is shown as "STARTED"