Occational client.transport.NoNodeAvailableException

IronMike · March 13, 2014, 6:02pm

I get this during bulk processing (see the Error in the middle of the log)

I tried it on my local machine, it doesn't happen, if I run it on some
remote EC2 instance, I do get the exception. Here is some output:

Settings settings = ImmutableSettings.settingsBuilder()

                                   .put("client.transport.sniff", true)

                                   .put("refresh_interval", "-1")  //Needs

to be updated after indexing is done

                                   .put("number_of_shards", 1)

                                   .put("number_of_replicas", "0")

                                   .put(

"index.merge.policy.merge_factor", 30)

                                   .put("client.transport.ping_timeout",

"10s") //didn't help for exception error

                                   //.put("discovery.type", "ec2")

                                   //.put("cloud.aws.access_key", "==")

                                   // .put("cloud.aws.secret_key",

"===")

                                   //.put("cloud.aws.region", "eu

-west-1")

                                   //.put("discovery.ec2.groups", "

elasticsearch")

                           .put("cluster.name", this.CLUSTER_NAME

).build();

Bulk Called: ID= 1, Actions=33, MB=5.461668

Bulk Called: ID= 2, Actions=29, MB=5.515789

Bulk Called: ID= 3, Actions=12, MB=5.691476

Bulk Called: ID= 4, Actions=17, MB=5.444193

Bulk Called: ID= 5, Actions=9, MB=5.492445

Bulk Called: ID= 6, Actions=17, MB=6.293821

Bulk Called: ID= 7, Actions=16, MB=5.769860

Bulk Called: ID= 8, Actions=18, MB=5.259554

Bulk Succeeded: ID= 1, took= 3471 ms

Bulk Called: ID= 9, Actions=11, MB=5.757115

Bulk Succeeded: ID= 2, took= 4207 ms

Bulk Called: ID= 10, Actions=13, MB=5.591648

Bulk Succeeded: ID= 3, took= 2453 ms

Bulk Called: ID= 11, Actions=13, MB=5.355669

Bulk Succeeded: ID= 4, took= 2568 ms

Bulk Called: ID= 12, Actions=16, MB=5.269400

Bulk Succeeded: ID= 5, took= 2233 ms

Bulk Called: ID= 13, Actions=14, MB=5.357479

Bulk Succeeded: ID= 6, took= 2434 ms

Bulk Called: ID= 14, Actions=18, MB=5.899198

Error: ID=7, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=11, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=12, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=8, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=13, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=14, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=9, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableError
: ID=10, org.elasticsearch.client.transport.NoNodeAvailableException: Nonode availableBulk
Called: ID= 15, Actions=20, MB=6.751245

Error: ID=15, org.elasticsearch.client.transport.NoNodeAvailableException:
No node availableError: ID=15, org.elasticsearch.client.transport.
NoNodeAvailableException: No node availableBulk Called: ID= 16, Actions=12,MB
=5.320580

Bulk Called: ID= 17, Actions=12, MB=5.31201

Bulk Called: ID= 18, Actions=15, MB=5.331088
Bulk Called: ID= 19, Actions=24, MB=5.307172

Bulk Called: ID= 20, Actions=32, MB=6.009871

Bulk Called: ID= 21, Actions=9, MB=6.020198

Bulk Called: ID= 22, Actions=1, MB=6.279574

Bulk Called: ID= 23, Actions=8, MB=5.452716

Bulk Called: ID= 24, Actions=19, MB=5.383386

Bulk Succeeded: ID= 16, took= 3604 ms

Bulk Called: ID= 25, Actions=22, MB=5.347178

Bulk Succeeded: ID= 17, took= 3926 ms

Bulk Called: ID= 26, Actions=58, MB=5.247683

Bulk Succeeded: ID= 18, took= 5014 ms

Bulk Called: ID= 27, Actions=90, MB=5.312535

Bulk Succeeded: ID= 19, took= 2253 ms

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5cf46b81-b2dc-4d73-b0a0-e3327e86050e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

IronMike · March 14, 2014, 7:15pm

Anyone ran into this problem? I am happening to see it when I process
larger documents. I am running one node, one shard.. I played with
HeapSize, but didn't matter.
In the example below, it missed 5 or 6 blocks, that's a lot of documents
that it didn't index. Any clues?

On Thursday, March 13, 2014 2:02:36 PM UTC-4, ZenMaster80 wrote:

I get this during bulk processing (see the Error in the middle of the log)

I tried it on my local machine, it doesn't happen, if I run it on some
remote EC2 instance, I do get the exception. Here is some output:

Settings settings = ImmutableSettings.settingsBuilder()
                                   .put("client.transport.sniff", true
)
                                   .put("refresh_interval", "-1")  //Needs 
to be updated after indexing is done
                                   .put("number_of_shards", 1)

                                   .put("number_of_replicas", "0")

                                   .put(
"index.merge.policy.merge_factor", 30)
                                   .put(
"client.transport.ping_timeout", "10s") //didn't help for exception error
                                   //.put("discovery.type", "ec2")

                                   //.put("cloud.aws.access_key", 
"==")
                                   // .put("cloud.aws.secret_key", 
"===")
                                   //.put("cloud.aws.region", "eu
-west-1")
                                   //.put("discovery.ec2.groups", "
elasticsearch")
                           .put("cluster.name", this.CLUSTER_NAME
).build();

Bulk Called: ID= 1, Actions=33, MB=5.461668

Bulk Called: ID= 2, Actions=29, MB=5.515789

Bulk Called: ID= 3, Actions=12, MB=5.691476

Bulk Called: ID= 4, Actions=17, MB=5.444193

Bulk Called: ID= 5, Actions=9, MB=5.492445

Bulk Called: ID= 6, Actions=17, MB=6.293821

Bulk Called: ID= 7, Actions=16, MB=5.769860

Bulk Called: ID= 8, Actions=18, MB=5.259554

Bulk Succeeded: ID= 1, took= 3471 ms

Bulk Called: ID= 9, Actions=11, MB=5.757115

Bulk Succeeded: ID= 2, took= 4207 ms

Bulk Called: ID= 10, Actions=13, MB=5.591648

Bulk Succeeded: ID= 3, took= 2453 ms

Bulk Called: ID= 11, Actions=13, MB=5.355669

Bulk Succeeded: ID= 4, took= 2568 ms

Bulk Called: ID= 12, Actions=16, MB=5.269400

Bulk Succeeded: ID= 5, took= 2233 ms

Bulk Called: ID= 13, Actions=14, MB=5.357479

Bulk Succeeded: ID= 6, took= 2434 ms

Bulk Called: ID= 14, Actions=18, MB=5.899198

Error: ID=7, org.elasticsearch.client.transport.NoNodeAvailableException:
No node availableError: ID=11, org.elasticsearch.client.transport.
NoNodeAvailableException: No node availableError: ID=12, org.elasticsearch
.client.transport.NoNodeAvailableException: No node availableError: ID=8,
org.elasticsearch.client.transport.NoNodeAvailableException: No node
availableError: ID=13, org.elasticsearch.client.transport.
NoNodeAvailableException: No node availableError: ID=14, org.elasticsearch
.client.transport.NoNodeAvailableException: No node availableError: ID=9,
org.elasticsearch.client.transport.NoNodeAvailableException: No node
availableError: ID=10, org.elasticsearch.client.transport.
NoNodeAvailableException: No node availableBulk Called: ID= 15, Actions=20
, MB=6.751245

Error: ID=15, org.elasticsearch.client.transport.NoNodeAvailableException:
No node availableError: ID=15, org.elasticsearch.client.transport.
NoNodeAvailableException: No node availableBulk Called: ID= 16, Actions=12
, MB=5.320580

Bulk Called: ID= 17, Actions=12, MB=5.31201

Bulk Called: ID= 18, Actions=15, MB=5.331088
Bulk Called: ID= 19, Actions=24, MB=5.307172

Bulk Called: ID= 20, Actions=32, MB=6.009871

Bulk Called: ID= 21, Actions=9, MB=6.020198

Bulk Called: ID= 22, Actions=1, MB=6.279574

Bulk Called: ID= 23, Actions=8, MB=5.452716

Bulk Called: ID= 24, Actions=19, MB=5.383386

Bulk Succeeded: ID= 16, took= 3604 ms

Bulk Called: ID= 25, Actions=22, MB=5.347178

Bulk Succeeded: ID= 17, took= 3926 ms

Bulk Called: ID= 26, Actions=58, MB=5.247683

Bulk Succeeded: ID= 18, took= 5014 ms

Bulk Called: ID= 27, Actions=90, MB=5.312535

Bulk Succeeded: ID= 19, took= 2253 ms

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/324002a9-ac0c-4d8f-b954-99d46265e4f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 14, 2014, 7:58pm

I'm curious, is there anything else in the es log files? Also are you
running on EC2 micro instances?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/61021936-8052-49b8-bcbf-495edfb173df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

IronMike · March 14, 2014, 8:20pm

I will post logs in a bit. I plan to wun on EC2, but currently just running
on a local machine i7, 4G Ram.

I had int concurrentRequests = Runtime.getRuntime().availableProcessors();
(Returns 8),
If I change this value to just "1", I don't get the exception, but indexing
performance slows down considerably. I am not sure if 8 requests is really
overwhelming the node.

On Friday, March 14, 2014 3:58:21 PM UTC-4, Binh Ly wrote:

I'm curious, is there anything else in the es log files? Also are you
running on EC2 micro instances?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1325173c-230d-416a-8a25-3b2201fa987a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 15, 2014, 10:44am

I am not able to examine your bulk code. From the output "no node
available" it seems you overwhelmed the cluster, yes. Testing bulk feeds on
a local machine only is kind of limited.

You have 5m/sec which is quite much for a small local ES cluster (without
knowing your data I can't confirm though, simple log data would make it).
Reset bulk to 1m/sec and see how it goes.

BTW your client settings contain some misunderstandings. You can't disable
the refresh rate there and you can't set merge factors. This must be set on
server side.

Jörg

On Fri, Mar 14, 2014 at 9:20 PM, ZenMaster80 sabdalla80@gmail.com wrote:

I will post logs in a bit. I plan to wun on EC2, but currently just
running on a local machine i7, 4G Ram.

I had int concurrentRequests = Runtime.getRuntime().availableProcessors();
(Returns 8),
If I change this value to just "1", I don't get the exception, but
indexing performance slows down considerably. I am not sure if 8 requests
is really overwhelming the node.

On Friday, March 14, 2014 3:58:21 PM UTC-4, Binh Ly wrote:

I'm curious, is there anything else in the es log files? Also are you
running on EC2 micro instances?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1325173c-230d-416a-8a25-3b2201fa987a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/1325173c-230d-416a-8a25-3b2201fa987a%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH10u6yLdPRp-xiP6V7Xn7SRgOMzmzAptijbmzwXtJ_kQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.