App hangs (with es blocking requests)

Elastic Stack Elasticsearch
1

Hi all,

I've been experiencing a bug where my application hangs. I haven't been
able to work out the problem and the code looks okay. Inspecting the
threads I see several are blocked with the following:

  • sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
    may be imprecise)
  • java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
    @bci=20, line=226 (Compiled frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTra
nsferQueue$Node,
org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node,
java.lang.Object, boolean, long) @bci=180, line=7
02 (Compiled frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object,
boolean, int, long) @bci=286, line=615 (Compil
ed frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled
frame)

  • java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068
    (Compiled frame)

java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1130 (Compiled frame)

  • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
    (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

Anyone know whether this is normal?

Thanks,

Nic

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Are you sure these threads are blocked?

From my understanding, this is just a thread in an ES scaling queue waiting
for a job.

You might want to play with thread pool settings to find out more about ES
threads waiting for jobs:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Jörg,

thanks for the reply. They appear as blocked when using jstack, but I don't
know if that is normal or not. Certainly, my application hangs without
throwing an exception. And the code itself makes a (blocking) serch request
to ES (using the java-library). It looks like this:

val search =

client.prepareSearch(indexName).setFilter(idsFilter(takedownTypes:
_*).addIds(id))
val searchResponse = search.execute().actionGet(2000)

I can't see anything obviously wrong with this, but perhaps I'm missing
something? Or it might be a problem elsewhere of course (although I've
tried my best to eliminate that possibility).

On Monday, 7 October 2013 13:02:42 UTC+1, nicola...@guardian.co.uk wrote:

Hi all,

I've been experiencing a bug where my application hangs. I haven't been
able to work out the problem and the code looks okay. Inspecting the
threads I see several are blocked with the following:

  • sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
    may be imprecise)
  • java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
    long) @bci=20, line=226 (Compiled frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTra
nsferQueue$Node,
org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node,
java.lang.Object, boolean, long) @bci=180, line=7
02 (Compiled frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object,
boolean, int, long) @bci=286, line=615 (Compil
ed frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled
frame)

  • java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068
    (Compiled frame)

java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1130 (Compiled frame)

  • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
    (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

Anyone know whether this is normal?

Thanks,

Nic

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Maybe it helps if you can post/gist the complete thread dump somewhere so
we can look at it?

Jörg
Am 08.10.2013 13:43 schrieb nicolas.long@guardian.co.uk:

Hi Jörg,

thanks for the reply. They appear as blocked when using jstack, but I
don't know if that is normal or not. Certainly, my application hangs
without throwing an exception. And the code itself makes a (blocking) serch
request to ES (using the java-library). It looks like this:

val search =

client.prepareSearch(indexName).setFilter(idsFilter(takedownTypes:
_*).addIds(id))
val searchResponse = search.execute().actionGet(2000)

I can't see anything obviously wrong with this, but perhaps I'm missing
something? Or it might be a problem elsewhere of course (although I've
tried my best to eliminate that possibility).

On Monday, 7 October 2013 13:02:42 UTC+1, nicola...@guardian.co.uk wrote:

Hi all,

I've been experiencing a bug where my application hangs. I haven't been
able to work out the problem and the code looks okay. Inspecting the
threads I see several are blocked with the following:

  • sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
    may be imprecise)
  • java.util.concurrent.locks.**LockSupport.parkNanos(java.**lang.Object,
    long) @bci=20, line=226 (Compiled frame)
  • org.elasticsearch.common.util.concurrent.jsr166y.
    LinkedTransferQueue.awaitMatch(org.elasticsearch.
    common.util.concurrent.**jsr166y.LinkedTra
    nsferQueue$Node, org.elasticsearch.common.util.**concurrent.jsr166y.**LinkedTransferQueue$Node,
    java.lang.Object, boolean, long) @bci=180, line=7
    02 (Compiled frame)
  • org.elasticsearch.common.util.concurrent.jsr166y.
    LinkedTransferQueue.xfer(java.**lang.Object, boolean, int, long)
    @bci=286, line=615 (Compil
    ed frame)
  • org.elasticsearch.common.util.**concurrent.jsr166y.**LinkedTransferQueue.poll(long,
    java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled
    frame)
  • java.util.concurrent.**ThreadPoolExecutor.getTask() @bci=141,
    line=1068 (Compiled frame)
  • java.util.concurrent.ThreadPoolExecutor.runWorker(
    java.util.concurrent.**ThreadPoolExecutor$Worker) @bci=26, line=1130
    (Compiled frame)
  • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5,
    line=615 (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

Anyone know whether this is normal?

Thanks,

Nic

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Nicola,

This is normal. There are lot of thread pools in Elasticsearch and this one
is just waiting for something to do.

Can perhaps post a complete stack dump of all threads?

Also - do other search request hang as well or just this one?

Cheers,
Boaz

On Monday, October 7, 2013 2:02:42 PM UTC+2, nicola...@guardian.co.uk wrote:

Hi all,

I've been experiencing a bug where my application hangs. I haven't been
able to work out the problem and the code looks okay. Inspecting the
threads I see several are blocked with the following:

  • sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
    may be imprecise)
  • java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
    long) @bci=20, line=226 (Compiled frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTra
nsferQueue$Node,
org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node,
java.lang.Object, boolean, long) @bci=180, line=7
02 (Compiled frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object,
boolean, int, long) @bci=286, line=615 (Compil
ed frame)

org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled
frame)

  • java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068
    (Compiled frame)

java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1130 (Compiled frame)

  • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
    (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

Anyone know whether this is normal?

Thanks,

Nic

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.