We definitely use scan/scroll a lot. It primarily gets used as part of our indexing process. I would anticipate that we generally only get back a single page of results. I would expect the calls to scrollResponse.getHits to happen in fairly quick succession.
We typically use something that looks like the following:
TimeValue scrollKeepAlive = new TimeValue(60000);
SearchRequestBuilder searchRequest = client.client().prepareSearch(indices); // always either 1 or 2 indices
.setSearchType(SearchType.SCAN)
.setTypes(type)
.setScroll(scrollKeepAlive)
.setFilter(filter)
.setSize(500);
addFieldsToSearchRequest(searchRequest, fieldList);
org.elasticsearch.action.search.SearchResponse scrollResp = searchRequest.execute().actionGet();
if (scrollResp.status() != RestStatus.OK) {
String msg = "Error running query. Status=" + scrollResp.status();
throw new SearcherException(msg);
}
while (true) {
scrollResp = client.client().prepareSearchScroll(scrollResp.getScrollId()).setScroll(scrollKeepAlive).execute().actionGet();
if (scrollResp.status() != RestStatus.OK) {
String msg = "Error running query. Status=" + scrollResp.status();
throw new SearcherException(msg);
}
if (scrollResp.getHits().hits().length == 0)
break;
for (SearchHit hit : scrollResp.getHits()) {
// DO SOMETHING WITH THE HIT (that doesn’t take very long)
}
}
From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of simonw
Sent: Thursday, February 06, 2014 1:18 PM
To: elasticsearch@googlegroups.com
Subject: Re: ElasticSearch server lock up
Hey folks,
am I right that you both use scan / scroll? Can you tell me a bit about how you use it and if there could be long intervals between fetching results?
simon
On Thursday, February 6, 2014 9:34:45 PM UTC+1, simonw wrote:
thanks for the info though! Here is the pull request to throw an exception instead of going into an infinite loop
I assume you are not indexing into this embedded instance?
simon
On Thursday, February 6, 2014 9:23:19 PM UTC+1, Jay Modi wrote:
Hi Simon,
-
Yes we use the delete by query
-
We do not use the hasChild and hasParent queries but use filtered queries on parent ids like this:
QueryBuilder query = QueryBuilders.matchQuery( "someField", value );
FilterBuilder filter = FilterBuilders.boolFilter().must( FilterBuilders.termFilter( "_parent", id ) );
QueryBuilder filteredQuery = QueryBuilders.filteredQuery( query, filter );
-
No Exceptions in the logs
-
Yes we embed elasticsearch in our application (running as a single node)
-
No we cannot reproduce on demand; it has occurred during normal runtime of our application
-
We have seen the issue in 0.20.2 (JDK6) and 0.90.8 (JDK7u45)
-
No not forcefully flushing
On Thursday, February 6, 2014 2:47:38 PM UTC-5, simonw wrote:
Hey folks,
we are trying to get to the bottom of this and I'd want to have some more infos here. Can you provide us with more insight in what you are doing with ES. What I'd be interested in is:
- are you using delete by query
- are you running parent child queries
- are you seeing any exceptions in the logs
- are you embedding elasticsearch
- can you reproduce the problem and if so what are you doing to do so.
- can you tell me the latest version you have see this happening? I could see some problems in 0.90.7 that are fixed in 0.90.8 though.
- are you forcefully flushing ie. calling flush via the API or update engine settings of any kind?
simon
On Thursday, February 6, 2014 3:03:17 PM UTC+1, Jay Modi wrote:
Additionally, I took a heap dump from the ES 0.90.7 instance and I believe I see why the code was stuck in acquire.
I looked at all of the SearcherManager instance and the index readers since it looks like that is what the code is trying to increment a count in when it acquires one. There were 24 readers in total associated with SearcherManager instances. Every reader except one had a refCount of 1, one had a refCount of 0. Now if there is a race condition while the code is in acquire where the refCount goes down to 0 (reader closes), while the acquire method is being called the loop will run forever as tryIncRef in a IndexReader will always return false if the refCount is not greater than 0.
Hope that makes sense. FWIW my understanding of the code could be wrong and something else could be going on; if clarification or more information from the heap dump/jstack is needed, please let me know.
On Thursday, February 6, 2014 8:50:22 AM UTC-5, Jay Modi wrote:
No we are not running any plugins. Our application uses the Java API to query elasticsearch.
We also had this happen on another instance running ES 0.20.2 with JDK6. Restarting "fixes" it but we only run a single node so when this happens everything with our application stops working.
On Thursday, February 6, 2014 7:28:34 AM UTC-5, Adrien Grand wrote:
On Fri, Jan 24, 2014 at 5:17 PM, Jay Modi <jay...@gmail.commailto:jay...@gmail.com> wrote:
Coincidentally we hit a similar issue on Windows yesterday with 0.90.8 and JDK 7u45. I took a jstack dump and noticed that all of my search threads were in ReferenceManager.acquire(). Looking at the Lucene code for acquire there is a "busy wait" while it tries to increment a reference. The system has stayed in this state and subsequent jstack dumps show the same behavior with all search threads in acquire. Prior to the lockup a index had completed and the lockup with a few searches and count queries.
This looks bad. Are you using any particular plugin that may acquire/release searchers?
--
Adrien Grand
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/zK5pLxVeLJ4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3bf685c5-1abd-4a58-9341-3d19f57465f9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A23CDD02F91F7D4293CEA2CEA11E5BF20EDF4455%40005-TK5MPN2-021.MGDADSK.autodesk.com.
For more options, visit https://groups.google.com/groups/opt_out.