Elastic client threads hanging?

Elastic Stack Elasticsearch
1

Hi,

Our application that runs queries against an Elastic server got itself
into an apparent hanging state earlier today. We took a thread dump
and can see quite a few elastic threads. We have 4 threads executing
Elastic commands, they appear to be in an extended WAITING state, e.g.
"Thread t@553". We are not seeing any progress on these WAITING
threads - maybe we're missing some sort of timeout?

http://gist.github.com/2420058 has the dump & the bit of Java code we
are executing from the 4 client threads.

Thanks in advance for your help.

cheers
Mike

2

Sadly, the thread dump on the client side won't help, is there a chance for
a recreation?

On Thu, Apr 19, 2012 at 1:21 PM, Mike Hill mikewhill@gmail.com wrote:

Hi,

Our application that runs queries against an Elastic server got itself
into an apparent hanging state earlier today. We took a thread dump
and can see quite a few elastic threads. We have 4 threads executing
Elastic commands, they appear to be in an extended WAITING state, e.g.
"Thread t@553". We are not seeing any progress on these WAITING
threads - maybe we're missing some sort of timeout?

Elastic client hang · GitHub has the dump & the bit of Java code we
are executing from the 4 client threads.

Thanks in advance for your help.

cheers
Mike

3

ok, thanks - i don't think it's easily reproducable, but if it happens
again, what other info should we capture to help investigate?

-- mike

On Apr 19, 3:57 pm, Shay Banon kim...@gmail.com wrote:

Sadly, the thread dump on the client side won't help, is there a chance for
a recreation?

On Thu, Apr 19, 2012 at 1:21 PM, Mike Hill mikewh...@gmail.com wrote:

Hi,

Our application that runs queries against an Elastic server got itself
into an apparent hanging state earlier today. We took a thread dump
and can see quite a few elastic threads. We have 4 threads executing
Elastic commands, they appear to be in an extended WAITING state, e.g.
"Thread t@553". We are not seeing any progress on these WAITING
threads - maybe we're missing some sort of timeout?

http://gist.github.com/2420058has the dump & the bit of Java code we
are executing from the 4 client threads.

Thanks in advance for your help.

cheers
Mike

4

Because elasticsearch is evented by design, getting an understanding based
on things like thread dump and the like does not help much..., so a
recreation would be the best bet to get this nailed down as fast as
possible.

On Thu, Apr 19, 2012 at 6:16 PM, Mike Hill mikewhill@gmail.com wrote:

ok, thanks - i don't think it's easily reproducable, but if it happens
again, what other info should we capture to help investigate?

-- mike

On Apr 19, 3:57 pm, Shay Banon kim...@gmail.com wrote:

Sadly, the thread dump on the client side won't help, is there a chance
for
a recreation?

On Thu, Apr 19, 2012 at 1:21 PM, Mike Hill mikewh...@gmail.com wrote:

Hi,

Our application that runs queries against an Elastic server got itself
into an apparent hanging state earlier today. We took a thread dump
and can see quite a few elastic threads. We have 4 threads executing
Elastic commands, they appear to be in an extended WAITING state, e.g.
"Thread t@553". We are not seeing any progress on these WAITING
threads - maybe we're missing some sort of timeout?

http://gist.github.com/2420058has the dump & the bit of Java code we
are executing from the 4 client threads.

Thanks in advance for your help.

cheers
Mike