ES 1.7.3 Java API error when I close Node

Elastic Stack Elasticsearch
1

Hi folks,

I have an error when I close my node using Java API.

oct. 26, 2015 10:06:59 AM org.elasticsearch.node.internal.InternalNode stop
INFOS: [Melody Guthrie] stopping ...
oct. 26, 2015 10:06:59 AM org.elasticsearch.indices.cluster.IndicesClusterStateService sendFailShard
AVERTISSEMENT: [Melody Guthrie] [[sm9][4]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [sm9][4]: Recovery failed from [Apalla][dGMjbk1tSTadfn8ey7nwBA][PORW7_16000][inet[/10.80.62.14:9300]] into [Melody Guthrie][WoCjUrGSRgyyi6YsAD1-7g][PORW7_16000][inet[/10.80.62.14:9
301]]
        at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280)
        at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70)
        at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:567)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.TransportException: transport stopped, action: internal:index/shard/recovery/start_recovery
        at org.elasticsearch.transport.TransportService$2.run(TransportService.java:178)
        ... 3 more

oct. 26, 2015 10:06:59 AM org.elasticsearch.node.internal.InternalNode stop
INFOS: [Melody Guthrie] stopped
oct. 26, 2015 10:06:59 AM org.elasticsearch.node.internal.InternalNode close
INFOS: [Melody Guthrie] closing ...
oct. 26, 2015 10:06:59 AM org.elasticsearch.cluster.action.shard.ShardStateAction$1 handleException
AVERTISSEMENT: [Melody Guthrie] failed to send failed shard to [Apalla][dGMjbk1tSTadfn8ey7nwBA][PORW7_16000][inet[/10.80.62.14:9300]]
org.elasticsearch.transport.SendRequestTransportException: [Apalla][inet[/10.80.62.14:9300]][internal:cluster/shard/failure]
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:286)
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:249)
        at org.elasticsearch.cluster.action.shard.ShardStateAction.innerShardFailed(ShardStateAction.java:102)
        at org.elasticsearch.cluster.action.shard.ShardStateAction.shardFailed(ShardStateAction.java:89)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.sendFailShard(IndicesClusterStateService.java:878)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:870)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.handleRecoveryFailure(IndicesClusterStateService.java:825)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.access$300(IndicesClusterStateService.java:84)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService$PeerRecoveryListener.onRecoveryFailure(IndicesClusterStateService.java:819)
        at org.elasticsearch.indices.recovery.RecoveryStatus.fail(RecoveryStatus.java:175)
        at org.elasticsearch.indices.recovery.RecoveriesCollection.failRecovery(RecoveriesCollection.java:122)
        at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:280)
        at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:70)
        at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:567)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:270)
        ... 17 more

oct. 26, 2015 10:06:59 AM org.elasticsearch.node.internal.InternalNode close
INFOS: [Melody Guthrie] closed
[success] Total time: 8 s, completed 26 oct. 2015 10:06:59
>

I use the following code to connect/disconnect:

Node node = nodeBuilder().clusterName("my-testing-cluster").node();
Client client = node.client();
[...]
node.close();

Any idea?

2

Can you explain why you are you doing this?

Basically told ES to shutdown, and it's trying to transfer shards off this node but it can't. Hence the error.

3

I make a web interface to search trough the ES index.

My goal is basically to connect to ES master node, which is in my local machine in developpement environment. I try to instanciate a new node and proceed a query as written in documentation but I have this error when I try to close/shutdown the node.

Finally I use the Transport Client method which is much more efficient in my case.

4

As @warkolm already pointed out, there seems to be some shard transfer going on to or from the node you are shutting down. This seems odd since you mentioned you are using a Transport Client (which is good for just issuing some queries) which by its nature shouldn't join the cluster. So there should not be any recovery going on. From the snippet you show it looks like you might be using the Node Client, in which case you should make it only join as a client, without shards being allocated to it (either by setting the node.data setting to false or the node.client to true). This can also be done using NodeBuilder.data(false) or NodeBuilder.client(true).

5

Thank you for your answers. I don't use the Node Client method anymore, that solved my problem.