Node Client with bulk request indefinitely blocked thread when ClusterBlockException is being thrown


(Cosmin Vasii) #1

I am using the dataless NodeClient to connect to my cluster (version is 1.1.1). Everything is working ok, except when failures occur. The scenario is the following:
-I have an application java based which connects to ES Cluster (application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But my call never catches the exception:

Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

My code is something like this

    BulkResponse response;
    try {
        response = requestBuilder.execute().actionGet();
    }
    catch(NoNodeAvailableException ex){
        LOGGER.error("Cannot connect to ES Cluster:  " + ex.getMessage());
        throw ex;
    }
    catch (ClusterBlockException ex){
        LOGGER.error("Cannot connect to ES Cluster:  " + ex.getMessage());
        throw ex;
    }
    catch (Exception ex) {

        LOGGER.error("Exception in processing indexing request by ES server. " + ex.getMessage());
    }

When I use a single request everything is ok. I also noticed a TODO in the ES code in the TransportBulkAction.java

private void executeBulk(final BulkRequest bulkRequest, final long startTime, final ActionListener listener, final AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
clusterState.blocks().globalBlockedRaiseException(ClusterBlockLevel.WRITE);

    ....}

Is this a known situation or a known bug?


(system) #2