Cluster does not send data to slave

(Mike) #1

Hi all,

i've setup a cluster with two nodes and the slave does not receive any data.

The master finds the slave and vice versa, but the slave reports back

[2015-11-19 13:42:56,540][DEBUG][action.admin.indices.stats] [Arturo Falcones] [indices:monitor/stats] failed to execute operation for shard [[logstash-2015.11.19][4], node[43hahGbBQC-XsBux2YfFgA], [R], v[5], s[IN
ITIALIZING], a[id=kBYTSXkqQEGjpKlBWEuw3g], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-11-19T13:40:56.591Z]]]
[logstash-2015.11.19][[logstash-2015.11.19][4]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IndexNotFoundException[no such index];
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Caused by: [logstash-2015.11.19] IndexNotFoundException[no such index]
        at org.elasticsearch.indices.IndicesService.indexServiceSafe(
        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(
        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(
        ... 7 more

I'm trying to find what that error means, but im not having any luck.

(Mark Walkom) #2

What version are you on?

(Mike) #3

i was on 1.6 but then i upgrade the whole cluster to 2.0 and it just worked.

(Sumesh Agarwal) #4

I am facing a similar issue, but I am already on ES 2.0. The weird thing is relocations fails only in a particular situation. More details -
3 nodes with index data in attached EBS volumes - relocations work like a charm!
I created a new ec2 instance with local storage, within the same VPC and belonging to the same security group. This node attaches successfully with the cluster, and initiates relocation of a couple of shards (monitoring via kopf plugin), and this gets stuck!
I get the same exception as above. The data path for this node created this - "/nodes/0/node.lock", so it cannot be a permissions issue. I tried restarting the new node, but no luck.
Any help please?

(Mark Walkom) #5

This seems to be similar, but different enough to warrant your own thread.

(Sumesh Agarwal) #6

@warkolm Thank you, I have initiated a new thread for this. Please find it here.

(system) #7