Cluster does not send data to slave

Elastic Stack Elasticsearch
1

Hi all,

i've setup a cluster with two nodes and the slave does not receive any data.

The master finds the slave and vice versa, but the slave reports back

[2015-11-19 13:42:56,540][DEBUG][action.admin.indices.stats] [Arturo Falcones] [indices:monitor/stats] failed to execute operation for shard [[logstash-2015.11.19][4], node[43hahGbBQC-XsBux2YfFgA], [R], v[5], s[IN
ITIALIZING], a[id=kBYTSXkqQEGjpKlBWEuw3g], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-11-19T13:40:56.591Z]]]
[logstash-2015.11.19][[logstash-2015.11.19][4]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IndexNotFoundException[no such index];
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:399)
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:376)
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:365)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:299)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: [logstash-2015.11.19] IndexNotFoundException[no such index]
        at org.elasticsearch.indices.IndicesService.indexServiceSafe(IndicesService.java:295)
        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:97)
        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:395)
        ... 7 more

I'm trying to find what that error means, but im not having any luck.
Cheers

2

What version are you on?

3

i was on 1.6 but then i upgrade the whole cluster to 2.0 and it just worked.

4

I am facing a similar issue, but I am already on ES 2.0. The weird thing is relocations fails only in a particular situation. More details -
3 nodes with index data in attached EBS volumes - relocations work like a charm!
I created a new ec2 instance with local storage, within the same VPC and belonging to the same security group. This node attaches successfully with the cluster, and initiates relocation of a couple of shards (monitoring via kopf plugin), and this gets stuck!
I get the same exception as above. The data path for this node created this - "/nodes/0/node.lock", so it cannot be a permissions issue. I tried restarting the new node, but no luck.
Any help please?

5

This seems to be similar, but different enough to warrant your own thread.

6

@warkolm Thank you, I have initiated a new thread for this. Please find it here.