Ok,.. I've turned on DEBUG log level, and now I see this:
DEBUG threadpool - [Boneyard] creating thread_pool [cached], type [cached],
keep_alive [30s]
DEBUG threadpool - [Boneyard] creating thread_pool [index], type [cached],
keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [bulk], type [cached],
keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [search], type [cached],
keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [percolate], type
[cached], keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [management], type
[scaling], min [1], size [5], keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [flush], type [scaling],
min [1], size [10], keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [merge], type [scaling],
min [1], size [20], keep_alive [5m]
DEBUG threadpool - [Boneyard] creating thread_pool [refresh], type
[cached], keep_alive [1m]
DEBUG threadpool - [Boneyard] creating thread_pool [snapshot], type
[scaling], min [1], size [5], keep_alive [5m]
DEBUG netty - [Boneyard] using worker_count[8], port[9300-9400],
bind_host[null], publish_host[null], compress[false], connect_timeout[30s],
connections_per_node[2/4/1]
DEBUG transport - [Boneyard] node_sampler_interval[5s]
DEBUG NioProviderMetadata - Using the autodetected NIO constraint level: 0
DEBUG netty - [Boneyard] Connected to node [[#transport#-1][inet[{SERVER
URL}/{SERVER IP}:9300]]]
All cyber users: 998
INFO transport - [Boneyard] failed to get node info for [#transport#-1][{SERVER
URL}/{SERVER IP}:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [inet[{SERVER
URL}/{SERVER IP}:9300]][cluster/nodes/info] request_id [104] timed out
after [5002ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(
TransportService.java:347)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
DEBUG netty - [Boneyard] Disconnected from [[#transport#-1][inet[{SERVER
URL}/{SERVER IP}:9300]]]
Exception in thread "main"
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(
TransportClientNodesService.java:214)
at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(
TransportActionNodeProxy.java:77)
at org.elasticsearch.transport.TransportService$Adapter$2$1.run(
TransportService.java:310)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Before I've got this exception I was able to run few searches and several
update.
Looks like this is just a timeout. But I don't see any problem with network
connection.
I thought that the server is very busy and couldn't return the result in 5
seconds. But I've checked other applications which use the same ES, and it
looks like they are in normal operation, and return results relatively
quickly. Even this application sometimes works just fine, but some times
gives this exception.
Here is a little of stats from ES:
When I've got the exception I had:
Search requests per second - 200
Search time per second - 2,000
Get requests per second - 10
Get time per second - 100
I could debug more, but I need some direction, what should I do next.
Please help,
Thank you,
Eugene S.
On Saturday, April 21, 2012 7:21:32 AM UTC-4, kimchy wrote:
If they are on the same box, it just means that they use the loopback
interface, but its still network/sockets/... .
On Thu, Apr 19, 2012 at 5:58 PM, Eugene Strokin eugene@strokin.infowrote:
Thank you for the reply. That was my initial thought as well. But the
thing is, that the server which connects to ES and Elasticsearch are on the
same physical box. So, there is any network connection between them which
could be lost (I should mentioned it right off).
Does it change anything?
Anyway, I'll try to turn on logging on org.elasticsearch.client.**transport
and see what it says.
Thanks again.
On Thursday, April 19, 2012 10:52:38 AM UTC-4, kimchy wrote:
This means that you started to get disconnections between the client
(transport) and the server. It will try and reconnect automatically, and
possibly manages to do it. For more information, you can turn on logging on
org.elasticsearch.client.**transport.
On Wed, Apr 18, 2012 at 5:17 PM, Eugene Strokin eugene@strokin.infowrote:
After several months of work with no problem (without even restart), my
application start to throw this exception once in a while:
Exception in thread "main" org.elasticsearch.client.transport.
NoNodeAvailableException: No node available
at org.elasticsearch.client.transport.TransportClientNodesService$
*RetryListener.onFailure(TransportClientNodesService.java:214*)
at org.elasticsearch.action.TransportActionNodeProxy$1.
handleException(TransportActio**nNodeProxy.java:77)
at org.elasticsearch.transport.**TransportService$Adapter$2$1.**run(
TransportService.java:310)
at java.util.concurrent.**ThreadPoolExecutor$Worker.**runTask(
ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:680)
And it continues to work with no problem.
I couldn't find any clues why it could be happening.
Does anybody had this problem? Or could someone give some ideas how to
investigate this?
Thanks in advance
Eugene S.
