Node death on load

[philv (Phil)]

I have a small 3-node cluster, I try loading 180 million documents to is using the Java TransportClient and the load I send the documents to always dies. The ES process ends. All I can see in the logs is a lot of GC time percentage wise. My hypothesis is that this one node is doing all the routing and I was wondering whether I have to set any other settings to allow the client itself to route between different nodes during a load. I am using the BulkLoader and this is how I configure a TransportClient:

  public static Client getExternalClient(String ip) throws AnException {
    InetAddress iaddress;
    try {
        iaddress = InetAddress.getByName(ip);
    } catch (UnknownHostException e) {
        throw new AnException("Cannot get a Client to ES cluster, perhaps the host name / IP address is incorrect", e);
    }

    return new PreBuiltTransportClient(getSettingsForExternalClient())
            .addTransportAddress(new TransportAddress(iaddress, 9300));

}

public static Settings getSettingsForExternalClient() {
    Settings.Builder b = Settings.builder()
            .put("cluster.name", Constants.getClusterName())
            .put("client.transport.nodes_sampler_interval", "30s")
            .put("client.transport.ping_timeout", "30s");
    return b.build();
}

I call getExternalClient( ip)
Where ip is the ES node that always dies.

I'd noticed here https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html that setting:

.put("client.transport.sniff", true)

seems like it might be the thing to do, but I thought I'd check in here and make sure I am not missing anything.

Cheers,
Phil