Randomly no response for search query

Timmy_Ziesenhenne · April 28, 2014, 10:03am

We use elasticsearch since a couple of weeks and there are over 160 million
of entries separated in a couple of indexes splitted by month (e.g.
index_2014_01,index_2014_02,....).
Sometimes if we send a bulk query requests to the elasticsearch nodes for
some requests there is no responses and no timeouts. There is no chance to
reproduce this problem because it happens randomly and also on querys wich
works seconds before with a minimum response time. We started to debug down
to the elasticsearch transport client without success. We noticed that
there was a request without response (Breakpoints where hooked in
response-listener, request-listener and failure-listener) - even after a
long wait time there was no response or failure. There is no exception or
log-output which indicates that problem except that after a time there is a
"disconnected from node" message.
There is not much load on the elasticsearch cluster and on our firewall all
ports from 9200 to 9400 are open

Our application structure:
TOMCAT 7 (JAVAEE-7u25) --> spring-data-elasticsearch-1.0.0.M2 -->
elasticsearch-1.1.0(1.0.0 with same issue) ----------> elasticsearch (8
nodes) version 1.0.0 (JAVA-7u25)

This issue happens randomly and we have no idea where it comes from.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ae9c972b-6fae-4cf2-8a0c-d43143a09ac2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

spinscale · May 5, 2014, 10:10am

Hey,

very hard to tell what has been happening here without more information.
There has been a fix for this added recently, see

github.com/elastic/elasticsearch

Search might not return on thread pool rejection

committed 07:25AM - 05 May 14 UTC

kimchy

+148 -17

When a thread pool rejects the execution on the local node, the search might not… return. This happens due to the fact that we move to the next shard only *within* the execution on the thread pool in the start method. If it fails to submit the task to the thread pool, it will go through the fail shard logic, but without "counting" the current shard itself. When this happens, the relevant shard will then execute more times than intended, causing the total opes counter to skew, and for example, if on another shard the search is successful, the total ops will be incremented *beyond* the expectedTotalOps, causing the check on == as the exit condition to never happen. The fix here makes sure that the shard iterator properly progresses even in the case of rejections, and also includes improvement to when cleaning a context is sent in case of failures (which were exposed by the test). Though the change fixes the problem, we should work on simplifying the code path considerably, the first suggestion as a followup is to remove the support for operation threading (also in broadcast), and move the local optimization execution to SearchService, this will simplify the code in different search action considerably, and will allow to remove the problematic #firstOrNull method on the shard iterator. The second suggestion is to move the optimization of local execution to the TransportService, so all actions will not have to explicitly do the mentioned optimization. fixes #4887

However this only happens when you get rejected requests in your thread
pools, is this the case? You can check with the nodes stats API.

--Alex

On Mon, Apr 28, 2014 at 12:03 PM, Timmy Ziesenhenne <
timmy.ziesenhenne@gmail.com> wrote:

We use elasticsearch since a couple of weeks and there are over 160
million of entries separated in a couple of indexes splitted by month (e.g.
index_2014_01,index_2014_02,....).
Sometimes if we send a bulk query requests to the elasticsearch nodes for
some requests there is no responses and no timeouts. There is no chance to
reproduce this problem because it happens randomly and also on querys wich
works seconds before with a minimum response time. We started to debug down
to the elasticsearch transport client without success. We noticed that
there was a request without response (Breakpoints where hooked in
response-listener, request-listener and failure-listener) - even after a
long wait time there was no response or failure. There is no exception or
log-output which indicates that problem except that after a time there is a
"disconnected from node" message.
There is not much load on the elasticsearch cluster and on our firewall
all ports from 9200 to 9400 are open

Our application structure:
TOMCAT 7 (JAVAEE-7u25) --> spring-data-elasticsearch-1.0.0.M2 -->
elasticsearch-1.1.0(1.0.0 with same issue) ----------> elasticsearch (8
nodes) version 1.0.0 (JAVA-7u25)

This issue happens randomly and we have no idea where it comes from.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ae9c972b-6fae-4cf2-8a0c-d43143a09ac2%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/ae9c972b-6fae-4cf2-8a0c-d43143a09ac2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-udpsEDHXpSb%3DKAB6y1y31dwT5eSz9jbFhk5Z-zmhq3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.