Transport client ping timeout + no node available exception

I am using java transport client and it frequently gets no node
available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)
~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)
~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)
~[elasticsearch-0.90.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
~[na:1.6.0_33]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only
solution to increase the ping timeout?

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello,

i used the transport client in two projects(small size, below 100 GB and
below 2000 users, only two instances running with replication) and i never
had a NodeNotAvailableException(1 year running).
Do you have problems in your network? Is your es-server sometimes down?
I wouldn't increase the ping timeout, i would investigate why there isn't a
response after 120s.

Am Samstag, 15. Juni 2013 22:21:39 UTC+2 schrieb Abhishek Sanoujam:

I am using java transport client and it frequently gets no node
available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)

~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)

~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)

~[elasticsearch-0.90.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

~[na:1.6.0_33]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only
solution to increase the ping timeout?

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

My opinion is that it is relative to the other thread you had just open.

During long GC the JVM is somehow suspended. So your client can not see it anymore.

My 2 cents.

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 juin 2013 à 22:21, Abhishek Sanoujam abhi.sanoujam@gmail.com a écrit :

I am using java transport client and it frequently gets no node available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: No node available
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246) ~[elasticsearch-0.90.0.jar:na]
at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84) ~[elasticsearch-0.90.0.jar:na]
at org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305) ~[elasticsearch-0.90.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet below).

ImmutableSettings.settingsBuilder().put("cluster.name", clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only solution to increase the ping timeout?

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I'm running it in ec2 - both client and server, don't think there's been
any network issues.

I've been running other operations - reads, indexes etc, but haven't
seen the exception at those times.
The thing that I suspect here is the one-by-one indexing, i have 10
threads doing it - continuously (needs to update about 9m docs). And the
box in which elasticsearch server running is also running other
services, and the cpu usage is a bit high ~60-70% usage. Can it be
somehow related?

To answer your "Is your es-server sometimes down?" - i posted another
mail in this list - i saw that my es server died once, don't have any
clue how/why. That explains the node unavailable exception, but I saw
this even when the server is running.

Will setting up a replica help here? I'm running just one es server
(this is not production, and we'll be definitely having replicas in prod)

On 6/16/13 2:30 AM, cthoma wrote:

Hello,

i used the transport client in two projects(small size, below 100 GB
and below 2000 users, only two instances running with replication) and
i never had a NodeNotAvailableException(1 year running).
Do you have problems in your network? Is your es-server sometimes down?
I wouldn't increase the ping timeout, i would investigate why there
isn't a response after 120s.

Am Samstag, 15. Juni 2013 22:21:39 UTC+2 schrieb Abhishek Sanoujam:

I am using java transport client and it frequently gets no node
available exception:

Caused by:
org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
     at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)

~[elasticsearch-0.90.0.jar:na]
     at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)

~[elasticsearch-0.90.0.jar:na]
     at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)

~[elasticsearch-0.90.0.jar:na]
     at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

~[na:1.6.0_33]
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

~[na:1.6.0_33]
     ... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name
<http://cluster.name>",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the
only
solution to increase the ping timeout?

-- 
---------
Cheers,
Abhishek

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If the client is the problem, Davids post, it would not help. You have to
fix your client problem.
If the server is the problem, a replica can help. 9m docs for one server,
if one doc is not big, is enough, i think. But if you have large docs, i
would
use a second/third... server.

Am Samstag, 15. Juni 2013 23:12:07 UTC+2 schrieb Abhishek Sanoujam:

I'm running it in ec2 - both client and server, don't think there's been
any network issues.

I've been running other operations - reads, indexes etc, but haven't seen
the exception at those times.
The thing that I suspect here is the one-by-one indexing, i have 10
threads doing it - continuously (needs to update about 9m docs). And the
box in which elasticsearch server running is also running other services,
and the cpu usage is a bit high ~60-70% usage. Can it be somehow related?

To answer your "Is your es-server sometimes down?" - i posted another mail
in this list - i saw that my es server died once, don't have any clue
how/why. That explains the node unavailable exception, but I saw this even
when the server is running.

Will setting up a replica help here? I'm running just one es server (this
is not production, and we'll be definitely having replicas in prod)

On 6/16/13 2:30 AM, cthoma wrote:

Hello,

i used the transport client in two projects(small size, below 100 GB and
below 2000 users, only two instances running with replication) and i never
had a NodeNotAvailableException(1 year running).
Do you have problems in your network? Is your es-server sometimes down?
I wouldn't increase the ping timeout, i would investigate why there isn't
a response after 120s.

Am Samstag, 15. Juni 2013 22:21:39 UTC+2 schrieb Abhishek Sanoujam:

I am using java transport client and it frequently gets no node
available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)

~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)

~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)

~[elasticsearch-0.90.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

~[na:1.6.0_33]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only
solution to increase the ping timeout?

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You could try to disable refresh while updating. It could help.

Why are you updating so many docs. Is it a one shot operation?
If it is, i would recommend to delete the old index and reindex from scratch.
Updating docs comes with a cost (Delete + Insert operation).

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 juin 2013 à 23:12, Abhishek Sanoujam abhi.sanoujam@gmail.com a écrit :

I'm running it in ec2 - both client and server, don't think there's been any network issues.

I've been running other operations - reads, indexes etc, but haven't seen the exception at those times.
The thing that I suspect here is the one-by-one indexing, i have 10 threads doing it - continuously (needs to update about 9m docs). And the box in which elasticsearch server running is also running other services, and the cpu usage is a bit high ~60-70% usage. Can it be somehow related?

To answer your "Is your es-server sometimes down?" - i posted another mail in this list - i saw that my es server died once, don't have any clue how/why. That explains the node unavailable exception, but I saw this even when the server is running.

Will setting up a replica help here? I'm running just one es server (this is not production, and we'll be definitely having replicas in prod)

On 6/16/13 2:30 AM, cthoma wrote:

Hello,

i used the transport client in two projects(small size, below 100 GB and below 2000 users, only two instances running with replication) and i never had a NodeNotAvailableException(1 year running).
Do you have problems in your network? Is your es-server sometimes down?
I wouldn't increase the ping timeout, i would investigate why there isn't a response after 120s.

Am Samstag, 15. Juni 2013 22:21:39 UTC+2 schrieb Abhishek Sanoujam:

I am using java transport client and it frequently gets no node
available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)
~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)
~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)
~[elasticsearch-0.90.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
~[na:1.6.0_33]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only
solution to increase the ping timeout?

--

Cheers,
Abhishek
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,
Abhishek

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

For this use case(deleting and reindexing), we have used an "A/B
switching".
The old index still keeps alive and searches, while updateing, are
performed against the old index(with alias).
After succesfully updating, alias moves to the newly created index and all
searches are executed against the new index.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

For this use case(deleting and reindexing), i recommend something like an
"A/B switching".
The old index still keeps alive and searches, while updateing, are
performed against the old index(with alias).
After succesfully updating, alias moves to the newly created index and all
searches are executed against the new index.

Am Samstag, 15. Juni 2013 23:43:32 UTC+2 schrieb David Pilato:

You could try to disable refresh while updating. It could help.

Why are you updating so many docs. Is it a one shot operation?
If it is, i would recommend to delete the old index and reindex from
scratch.
Updating docs comes with a cost (Delete + Insert operation).

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 juin 2013 à 23:12, Abhishek Sanoujam <abhi.s...@gmail.com<javascript:>>
a écrit :

I'm running it in ec2 - both client and server, don't think there's been
any network issues.

I've been running other operations - reads, indexes etc, but haven't seen
the exception at those times.
The thing that I suspect here is the one-by-one indexing, i have 10
threads doing it - continuously (needs to update about 9m docs). And the
box in which elasticsearch server running is also running other services,
and the cpu usage is a bit high ~60-70% usage. Can it be somehow related?

To answer your "Is your es-server sometimes down?" - i posted another mail
in this list - i saw that my es server died once, don't have any clue
how/why. That explains the node unavailable exception, but I saw this even
when the server is running.

Will setting up a replica help here? I'm running just one es server (this
is not production, and we'll be definitely having replicas in prod)

On 6/16/13 2:30 AM, cthoma wrote:

Hello,

i used the transport client in two projects(small size, below 100 GB and
below 2000 users, only two instances running with replication) and i never
had a NodeNotAvailableException(1 year running).
Do you have problems in your network? Is your es-server sometimes down?
I wouldn't increase the ping timeout, i would investigate why there isn't
a response after 120s.

Am Samstag, 15. Juni 2013 22:21:39 UTC+2 schrieb Abhishek Sanoujam:

I am using java transport client and it frequently gets no node
available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)

~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)

~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)

~[elasticsearch-0.90.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

~[na:1.6.0_33]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only
solution to increase the ping timeout?

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

+1
definitely the way to go.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 juin 2013 à 00:25, cthoma thch0014@gmail.com a écrit :

For this use case(deleting and reindexing), i recommend something like an "A/B switching".
The old index still keeps alive and searches, while updateing, are performed against the old index(with alias).
After succesfully updating, alias moves to the newly created index and all searches are executed against the new index.

Am Samstag, 15. Juni 2013 23:43:32 UTC+2 schrieb David Pilato:

You could try to disable refresh while updating. It could help.

Why are you updating so many docs. Is it a one shot operation?
If it is, i would recommend to delete the old index and reindex from scratch.
Updating docs comes with a cost (Delete + Insert operation).

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 juin 2013 à 23:12, Abhishek Sanoujam abhi.s...@gmail.com a écrit :

I'm running it in ec2 - both client and server, don't think there's been any network issues.

I've been running other operations - reads, indexes etc, but haven't seen the exception at those times.
The thing that I suspect here is the one-by-one indexing, i have 10 threads doing it - continuously (needs to update about 9m docs). And the box in which elasticsearch server running is also running other services, and the cpu usage is a bit high ~60-70% usage. Can it be somehow related?

To answer your "Is your es-server sometimes down?" - i posted another mail in this list - i saw that my es server died once, don't have any clue how/why. That explains the node unavailable exception, but I saw this even when the server is running.

Will setting up a replica help here? I'm running just one es server (this is not production, and we'll be definitely having replicas in prod)

On 6/16/13 2:30 AM, cthoma wrote:

Hello,

i used the transport client in two projects(small size, below 100 GB and below 2000 users, only two instances running with replication) and i never had a NodeNotAvailableException(1 year running).
Do you have problems in your network? Is your es-server sometimes down?
I wouldn't increase the ping timeout, i would investigate why there isn't a response after 120s.

Am Samstag, 15. Juni 2013 22:21:39 UTC+2 schrieb Abhishek Sanoujam:

I am using java transport client and it frequently gets no node
available exception:

Caused by: org.elasticsearch.client.transport.NoNodeAvailableException:
No node available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)
~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:84)
~[elasticsearch-0.90.0.jar:na]
at
org.elasticsearch.transport.TransportService$Adapter$2$1.run(TransportService.java:305)
~[elasticsearch-0.90.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
~[na:1.6.0_33]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
~[na:1.6.0_33]
... 1 common frames omitted

I have set the 'client.transport.ping_timeout' to 120seconds (snippet
below).

ImmutableSettings.settingsBuilder().put("cluster.name",
clusterName).put("client.transport.ping_timeout", "120s");

How can I make the client more resilient to ping timeouts? Is the only
solution to increase the ping timeout?

--

Cheers,
Abhishek
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,
Abhishek

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.