Cluster issue -> raiseTimeoutFailure

Ciprian_2 · March 20, 2015, 12:59pm

Hi,

I have a java application that is indexind data in an Elasticsearch
cluster(3 nodes). The ES is well configured and is working ok(indexing
the received data from java).
Cluster configuration for each node from
/etc/elasticsearch/elasticsearch.yml

ES_MAX_MEM: 2g
ES_MIN_MEM: 2g
bootstrap:
mlockall: true
cluster:
name: clusterName
discovery:
zen:
ping:
multicast:
enabled: false
unicast:
hosts:
- elasticsearch-test-2-node-1
- elasticsearch-test-2-node-2
- elasticsearch-test-2-node-3
http:
max_initial_line_length: 48k
index:
number_of_replicas: 2
number_of_shards: 6
node:
name: elasticsearch-test-2-node-3
threadpool:
index:
type: fixed
size: 6
queue_size: 1500
search:
type: fixed
size: 6
queue_size: 1200

When I'm connecting the Es cluster(from java), I specify all the nodes :
node1, node2, node3.

The issue is appearing when I stop the 2 data nodes one by one(stop the
elasticsearch). In this case the cluster health is yellow and i can see the
remained master node(using "head" plugin). The master has now all the
primary shards. The replicas are "Unassigned". But the java application is
not indexing any more the data. The next exception appear on java :

org.elasticsearch.action.UnavailableShardsException: [indexName][2] [3]
shardIt, [1] active : Timeout waiting for [1m], request: index
{[indexName][typeName][Id], source[{ ..... }]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:491)
~[elasticsearch-1.1.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_51]

Shouldn't work properly the indexing in this case even with only the
master? *

If I am going to kill also the master the next logical exception appears
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:263)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:231)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.client.support.AbstractClient.update(AbstractClient.java:107)
~[elasticsearch-1.1.0.jar:na]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/176b3f2e-9e18-4018-a4a9-46b009dfd3d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · March 20, 2015, 4:45pm

You shouldn't sent MIN and MAX just use ES_HEAP_SIZE and it will set both,
you shouldn't also change the threadpools unless you understand and are
aware of what they entail.

I think the problem here would most likely be write consistency -

On 20 March 2015 at 05:59, Alex turcanciprian87@gmail.com wrote:

Hi,

I have a java application that is indexind data in an Elasticsearch
cluster(3 nodes). The ES is well configured and is working
ok(indexing the received data from java).
Cluster configuration for each node from
/etc/elasticsearch/elasticsearch.yml

ES_MAX_MEM: 2g
ES_MIN_MEM: 2g
bootstrap:
mlockall: true
cluster:
name: clusterName
discovery:
zen:
ping:
multicast:
enabled: false
unicast:
hosts:
- elasticsearch-test-2-node-1
- elasticsearch-test-2-node-2
- elasticsearch-test-2-node-3
http:
max_initial_line_length: 48k
index:
number_of_replicas: 2
number_of_shards: 6
node:
name: elasticsearch-test-2-node-3
threadpool:
index:
type: fixed
size: 6
queue_size: 1500
search:
type: fixed
size: 6
queue_size: 1200

When I'm connecting the Es cluster(from java), I specify all the nodes :
node1, node2, node3.

The issue is appearing when I stop the 2 data nodes one by one(stop the
elasticsearch). In this case the cluster health is yellow and i can see the
remained master node(using "head" plugin). The master has now all the
primary shards. The replicas are "Unassigned". But the java application
is not indexing any more the data. The next exception appear on java :

org.elasticsearch.action.UnavailableShardsException: [indexName][2] [3]
shardIt, [1] active : Timeout waiting for [1m], request: index
{[indexName][typeName][Id], source[{ ..... }]}
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:548)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:538)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:491)
~[elasticsearch-1.1.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_51]

Shouldn't work properly the indexing in this case even with only the
master? *

If I am going to kill also the master the next logical exception
appears
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:263)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:231)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
~[elasticsearch-1.1.0.jar:na]
at
org.elasticsearch.client.support.AbstractClient.update(AbstractClient.java:107)
~[elasticsearch-1.1.0.jar:na]

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/176b3f2e-9e18-4018-a4a9-46b009dfd3d2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/176b3f2e-9e18-4018-a4a9-46b009dfd3d2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JdbyjGdvh619yHnDO5%2BMYeiM%2B9sLh4H%2B4xbEAeOxozg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.