# ES 1.3.4 scrolling never ends

**URL:** <https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578>\
**Category:** Elasticsearch\
**Created:** [November 5, 2014, 12:48pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578 "2014-11-05T12:48:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yarden\_Bar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yarden_bar/32/736_2.png) [@Yarden\_Bar](https://discuss.elastic.co/u/Yarden_Bar)\
**Post date:** [November 5, 2014, 12:48pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/1 "2014-11-05T12:48:46Z")

</div>

Hi all,

I'm encountering a strange behavior when executing a search-scroll on a  
single node of ES-1.3.4 with Java client.

The scenario is as follows:

1. Start a single node of version 1.3.4

2. Add snapshot repository pointing to version 1.1.1 snapshots

3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node

4. Execute search on an index with

5. client.prepareSearch("my\_index\*").setQuery(QueryBuilders.filteredQuery  
(QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter(  
QueryBuilders.queryString(s"$terms AND  
snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") )) )  
.addFields(OBFields.values.map(\_.toString).toList: \_\*).setSize(pageSize  
).addSort(OBFields.updateNo.toString, SortOrder.ASC)  
.setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet()

6. Execute the following search scroll  
client.prepareSearchScroll(scrollId).setScroll(TimeValue.timeValueMinutes  
(3)).execute().actionGet()

I have a loop iterating over #6, providing the same scrollId and checking  
for (result.getHits().getHits().legth == 0) to terminate.  
I keep getting the same result 'page' with the same amount of results.

Any Idea??

Thanks,  
Yarden

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be0f385c-9d46-492b-a818-9bb04c92b214%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be0f385c-9d46-492b-a818-9bb04c92b214%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\
**Post date:** [November 5, 2014, 3:08pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/2 "2014-11-05T15:08:06Z")

</div>

You need to get the scroll ID from each response and use that one in the  
subsequent scan search. You cannot simply reuse the same scroll ID.

Brian

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d1f23ca4-13e6-4d1e-ad01-2cbda2810c94%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d1f23ca4-13e6-4d1e-ad01-2cbda2810c94%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Yarden\_Bar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yarden_bar/32/736_2.png) [@Yarden\_Bar](https://discuss.elastic.co/u/Yarden_Bar)\
**Post date:** [November 5, 2014, 3:52pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/3 "2014-11-05T15:52:25Z")

</div>

I'll try that and report....

Thanks,  
Yarden

On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:

> Hi all,
> 
> I'm encountering a strange behavior when executing a search-scroll on a  
> single node of ES-1.3.4 with Java client.
> 
> The scenario is as follows:
> 
> 1. Start a single node of version 1.3.4
> 
> 2. Add snapshot repository pointing to version 1.1.1 snapshots
> 
> 3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
> 
> 4. Execute search on an index with
> 
> 5. client.prepareSearch("my\_index\*").setQuery(QueryBuilders.  
> filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter  
> (  
> QueryBuilders.queryString(s"$terms AND  
> snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") )) )  
> .addFields(OBFields.values.map(\_.toString).toList: \_\*).setSize(  
> pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)  
> .setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet()
> 
> 6. Execute the following search scroll  
> client.prepareSearchScroll(scrollId).setScroll(TimeValue.  
> timeValueMinutes(3)).execute().actionGet()
> 
> I have a loop iterating over #6, providing the same scrollId and checking  
> for (result.getHits().getHits().legth == 0) to terminate.  
> I keep getting the same result 'page' with the same amount of results.
> 
> Any Idea??
> 
> Thanks,  
> Yarden

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c658566c-f2e4-4020-bd14-08e413c81a9c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c658566c-f2e4-4020-bd14-08e413c81a9c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Yarden\_Bar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yarden_bar/32/736_2.png) [@Yarden\_Bar](https://discuss.elastic.co/u/Yarden_Bar)\
**Post date:** [November 9, 2014, 1:18pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/4 "2014-11-09T13:18:12Z")

</div>

Update:  
Only when I set the SearchType to something else than  
the QUERY\_AND\_FETCH the scroll success to finish.

Any idea why QUERY\_THEN\_FETCH(the default) brings me to an endless loop?

The full code is:

val client = ESClientFactory.createByNode(ESNode.Builder,cluster = "test\_acm\_es")

```
val query = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter(
  QueryBuilders.queryString("((market:2 AND feed:55) OR (market:2 AND feed:32)) AND snapshotNo:[12614 TO 12627]")))

var result: SearchResponse = client.prepareSearch("orderbook-2014.10.21")
  .setQuery(query)
  .addFields(OBFields.values.map(_.toString).toList: _*)
  .setSearchType(SearchType.QUERY_THEN_FETCH)
  .setSize(1000)
  .addSort(OBFields.updateNo.toString, SortOrder.ASC)
  .setScroll(new Scroll(TimeValue.timeValueMinutes(5)))
.execute().actionGet()

println(s"Result total hits:${result.getHits.totalHits()}")
println(s"Result hits:${result.getHits.getHits().length}")
  do {

```

// result = new SearchScrollRequestBuilder(client,result.getScrollId).setScroll(TimeValue.timeValueMinutes(2)).execute().actionGet()  
result = client.prepareSearchScroll(result.getScrollId).setScroll(TimeValue.timeValueMinutes(2)).execute().actionGet()  
println(s"Iteration=$itr, scrollResult=${result.getHits.getHits.length}")  
itr += 1

```
  } while (result.getHits.getHits.length != 0)

```

Thanks for any idea...  
Yarden

On Wednesday, November 5, 2014 5:52:25 PM UTC+2, Yarden Bar wrote:

> I'll try that and report....
> 
> Thanks,  
> Yarden
> 
> On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:
> 
> > Hi all,
> > 
> > I'm encountering a strange behavior when executing a search-scroll on a  
> > single node of ES-1.3.4 with Java client.
> > 
> > The scenario is as follows:
> > 
> > 1. Start a single node of version 1.3.4
> > 
> > 2. Add snapshot repository pointing to version 1.1.1 snapshots
> > 
> > 3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
> > 
> > 4. Execute search on an index with
> > 
> > 5. client.prepareSearch("my\_index\*").setQuery(QueryBuilders.  
> > filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.  
> > queryFilter(  
> > QueryBuilders.queryString(s"$terms AND  
> > snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") )) )  
> > .addFields(OBFields.values.map(\_.toString).toList: \_\*).setSize(  
> > pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)  
> > .setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet  
> > ()
> > 
> > 6. Execute the following search scroll  
> > client.prepareSearchScroll(scrollId).setScroll(TimeValue.  
> > timeValueMinutes(3)).execute().actionGet()
> > 
> > I have a loop iterating over #6, providing the same scrollId and checking  
> > for (result.getHits().getHits().legth == 0) to terminate.  
> > I keep getting the same result 'page' with the same amount of results.
> > 
> > Any Idea??
> > 
> > Thanks,  
> > Yarden

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6f1a1178-d4e5-41e9-a464-68c6e9204779%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f1a1178-d4e5-41e9-a464-68c6e9204779%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Yarden\_Bar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yarden_bar/32/736_2.png) [@Yarden\_Bar](https://discuss.elastic.co/u/Yarden_Bar)\
**Post date:** [November 10, 2014, 12:27pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/5 "2014-11-10T12:27:11Z")

</div>

One issue I identified is the heap size was too small for the query, I've  
increased the heap memory and the CircuitBreakerException stopped happening.

But the scrolling still returning the SAME result.

An updated code example is below:  
import org.elasticsearch.action.search.SearchType  
import org.elasticsearch.client.transport.TransportClient  
import org.elasticsearch.common.settings.ImmutableSettings  
import org.elasticsearch.common.transport.InetSocketTransportAddress  
import org.elasticsearch.common.unit.TimeValue  
import org.elasticsearch.index.query.{FilterBuilders, QueryBuilders}  
import org.elasticsearch.search.Scroll  
import org.elasticsearch.search.sort.SortOrder

val es\_settings = ImmutableSettings.settingsBuilder().put("transport.sniff",  
true).put("cluster.name", "test\_acm\_es").build()  
var client = new TransportClient(es\_settings).addTransportAddress(new  
InetSocketTransportAddress("myServer",9300))  
val query = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),  
FilterBuilders.queryFilter(  
QueryBuilders.queryString("((market:2 AND feed:55) OR (market:2 AND  
feed:32))")))  
var result = client.prepareSearch("orderbook-2014.11.03").setTypes(List(  
"level"):\_\*).setQuery(query).setSearchType(SearchType.DFS\_QUERY\_THEN\_FETCH).  
setSize(10000).addSort("updateNo", SortOrder.ASC).setScroll(new Scroll(  
TimeValue.timeValueMinutes(5))).get()  
var scrollId = ""  
var itr = 0  
do {  
scrollId = result.getScrollId  
result = client.prepareSearchScroll(scrollId).setScroll(TimeValue.  
timeValueMinutes(3)).get()  
println(s"Iteration=$itr, scrollResult=${result.getHits.getHits.length}")  
// println("------------------------------------")  
// result.getHits.getHits.foreach(h =\> println(h.getId))  
// println("------------------------------------")  
itr+=1  
} while (result.getHits.getHits.length != 0)

enabling the print block reveals that the searchHit array is the same for  
each iteration...

Thanks,  
Yarden

On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:

> Hi all,
> 
> I'm encountering a strange behavior when executing a search-scroll on a  
> single node of ES-1.3.4 with Java client.
> 
> The scenario is as follows:
> 
> 1. Start a single node of version 1.3.4
> 
> 2. Add snapshot repository pointing to version 1.1.1 snapshots
> 
> 3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
> 
> 4. Execute search on an index with
> 
> 5. client.prepareSearch("my\_index\*").setQuery(QueryBuilders.  
> filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter  
> (  
> QueryBuilders.queryString(s"$terms AND  
> snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") )) )  
> .addFields(OBFields.values.map(\_.toString).toList: \_\*).setSize(  
> pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)  
> .setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet()
> 
> 6. Execute the following search scroll  
> client.prepareSearchScroll(scrollId).setScroll(TimeValue.  
> timeValueMinutes(3)).execute().actionGet()
> 
> I have a loop iterating over #6, providing the same scrollId and checking  
> for (result.getHits().getHits().legth == 0) to terminate.  
> I keep getting the same result 'page' with the same amount of results.
> 
> Any Idea??
> 
> Thanks,  
> Yarden

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 10, 2014, 2:06pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/6 "2014-11-10T14:06:05Z")

</div>

You must initiate scan/scroll with search type SCAN. The scan/scroll  
pattern is like this

SearchRequest searchRequest = new  
SearchRequestBuilder(client).setQuery(QueryBuilders.matchAllQuery()).request();  
searchRequest.searchType(SearchType.SCAN).scroll(request.getTimeout());  
SearchResponse searchResponse = client.search(searchRequest).actionGet();  
// get total hits here before entering the loop  
while (searchResponse.getScrollId() != null) {  
searchResponse =  
client.prepareSearchScroll(searchResponse.getScrollId())

.setScroll(request.getTimeout()).execute().actionGet();  
long hits = searchResponse.getHits().getHits().length;  
// process hits of a scroll here

}

Jörg

On Mon, Nov 10, 2014 at 1:27 PM, Yarden Bar [ayash.jorden@gmail.com](mailto:ayash.jorden@gmail.com) wrote:

> One issue I identified is the heap size was too small for the query, I've  
> increased the heap memory and the CircuitBreakerException stopped happening.
> 
> But the scrolling still returning the SAME result.
> 
> An updated code example is below:  
> import org.elasticsearch.action.search.SearchType  
> import org.elasticsearch.client.transport.TransportClient  
> import org.elasticsearch.common.settings.ImmutableSettings  
> import org.elasticsearch.common.transport.InetSocketTransportAddress  
> import org.elasticsearch.common.unit.TimeValue  
> import org.elasticsearch.index.query.{FilterBuilders, QueryBuilders}  
> import org.elasticsearch.search.Scroll  
> import org.elasticsearch.search.sort.SortOrder
> 
> val es\_settings = ImmutableSettings.settingsBuilder().put(  
> "transport.sniff", true).put("cluster.name", "test\_acm\_es").build()  
> var client = new TransportClient(es\_settings).addTransportAddress(new  
> InetSocketTransportAddress("myServer",9300))  
> val query = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),  
> FilterBuilders.queryFilter(  
> QueryBuilders.queryString("((market:2 AND feed:55) OR (market:2 AND  
> feed:32))")))  
> var result = client.prepareSearch("orderbook-2014.11.03").setTypes(List(  
> "level"):\_\*).setQuery(query).setSearchType(SearchType.DFS\_QUERY\_THEN\_FETCH  
> ).setSize(10000).addSort("updateNo", SortOrder.ASC).setScroll(new Scroll(  
> TimeValue.timeValueMinutes(5))).get()  
> var scrollId = ""  
> var itr = 0  
> do {  
> scrollId = result.getScrollId  
> result = client.prepareSearchScroll(scrollId).setScroll(TimeValue.  
> timeValueMinutes(3)).get()  
> println(s"Iteration=$itr, scrollResult=${result.getHits.getHits.length}")  
> // println("------------------------------------")  
> // result.getHits.getHits.foreach(h =\> println(h.getId))  
> // println("------------------------------------")  
> itr+=1  
> } while (result.getHits.getHits.length != 0)
> 
> enabling the print block reveals that the searchHit array is the same for  
> each iteration...
> 
> Thanks,  
> Yarden
> 
> On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:
> 
> > Hi all,
> > 
> > I'm encountering a strange behavior when executing a search-scroll on a  
> > single node of ES-1.3.4 with Java client.
> > 
> > The scenario is as follows:
> > 
> > 1. Start a single node of version 1.3.4
> > 
> > 2. Add snapshot repository pointing to version 1.1.1 snapshots
> > 
> > 3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
> > 
> > 4. Execute search on an index with
> > 
> > 5. client.prepareSearch("my\_index\*").setQuery(QueryBuilders.  
> > filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.  
> > queryFilter(  
> > QueryBuilders.queryString(s"$terms AND snapshotNo:[${mdp.fromSnapshot}  
> > TO ${mdp.toSnapshot}]") )) )  
> > .addFields(OBFields.values.map(\_.toString).toList: \_\*).setSize(  
> > pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)  
> > .setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet  
> > ()
> > 
> > 6. Execute the following search scroll  
> > client.prepareSearchScroll(scrollId).setScroll(TimeValue.tim  
> > eValueMinutes(3)).execute().actionGet()
> > 
> > I have a loop iterating over #6, providing the same scrollId and checking  
> > for (result.getHits().getHits().legth == 0) to terminate.  
> > I keep getting the same result 'page' with the same amount of results.
> > 
> > Any Idea??
> > 
> > Thanks,  
> > Yarden
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGgEhDf210fHVx%2BNqj-qFc5xu32zTp9FkK3W1Dtpi%3DJgg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGgEhDf210fHVx%2BNqj-qFc5xu32zTp9FkK3W1Dtpi%3DJgg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Yarden\_Bar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yarden_bar/32/736_2.png) [@Yarden\_Bar](https://discuss.elastic.co/u/Yarden_Bar)\
**Post date:** [November 10, 2014, 2:12pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/7 "2014-11-10T14:12:10Z")

</div>

Hi Jorg,

I cant use scan type because I need the documents sorted ASC on a field, scan returns the documents in the order they indexed.

Thanks

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1eb0d4dd-1659-48a2-929b-194ebd531465%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1eb0d4dd-1659-48a2-929b-194ebd531465%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 10, 2014, 2:21pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/8 "2014-11-10T14:21:01Z")

</div>

Scan is not really the order the docs are indexed (it depends on how the  
index segments in the shards return the docs).

But anyway, you can not scroll over a sorted result set.

Jörg

On Mon, Nov 10, 2014 at 3:12 PM, Yarden Bar [ayash.jorden@gmail.com](mailto:ayash.jorden@gmail.com) wrote:

> Hi Jorg,
> 
> I cant use scan type because I need the documents sorted ASC on a field,  
> scan returns the documents in the order they indexed.
> 
> Thanks
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/1eb0d4dd-1659-48a2-929b-194ebd531465%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1eb0d4dd-1659-48a2-929b-194ebd531465%40googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEfbojPZCOesok%2B5jdkRu6Z5CExTDzAVnwTCZXveL7dHw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEfbojPZCOesok%2B5jdkRu6Z5CExTDzAVnwTCZXveL7dHw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\
**Post date:** [November 10, 2014, 5:06pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/9 "2014-11-10T17:06:52Z")

</div>

A while back, I wrote my own post-query response sorting so that I could  
handle cases that Elasticsearch didn't. One case was sorting a scan query.  
I used a Java TreeSet class and could also limit it to the top 'N'  
(configurable) items. It is very, very quick, pretty much adding no  
overhead to the existing scan logic. And it supports an arbitrarily complex  
compound sort key, much like an SQL ORDERBY statement; it's very easy to  
construct.

Probably not useful for a normal user query, but it is very useful for an  
ad-hoc query in which I wish to scan across an indeterminately large result  
set but still sort the results.

One of these days, it might make a good plug-in candidate. But I am not  
sure how to integrate it with the scan API, so for now it's just part of  
the Java client layer.

Brian

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/74e311f5-ae54-4da1-9369-567e7bf03272%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/74e311f5-ae54-4da1-9369-567e7bf03272%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Yarden\_Bar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yarden_bar/32/736_2.png) [@Yarden\_Bar](https://discuss.elastic.co/u/Yarden_Bar)\
**Post date:** [November 12, 2014, 6:30pm UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/10 "2014-11-12T18:30:49Z")

</div>

Finally the issue was solved.

I forgot to mention that I had a Logstash output connected and it's protocol  
[http://logstash.net/docs/1.4.2/outputs/elasticsearch#protocol](http://logstash.net/docs/1.4.2/outputs/elasticsearch#protocol) was set to  
'node', meaning that logstash was part of my cluster.  
Once I set the protocol to 'transport',scrolling was perfect!!

Credit to my team-leader for guidance....

Thanks everyone for your help  
Yarden

On Monday, November 10, 2014 7:06:52 PM UTC+2, Brian wrote:

> A while back, I wrote my own post-query response sorting so that I could  
> handle cases that Elasticsearch didn't. One case was sorting a scan query.  
> I used a Java TreeSet class and could also limit it to the top 'N'  
> (configurable) items. It is very, very quick, pretty much adding no  
> overhead to the existing scan logic. And it supports an arbitrarily  
> complex compound sort key, much like an SQL ORDERBY statement; it's very  
> easy to construct.
> 
> Probably not useful for a normal user query, but it is very useful for an  
> ad-hoc query in which I wish to scan across an indeterminately large result  
> set but still sort the results.
> 
> One of these days, it might make a good plug-in candidate. But I am not  
> sure how to integrate it with the scan API, so for now it's just part of  
> the Java client layer.
> 
> Brian

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/44b5c8db-6c09-4527-b440-09d01bde3588%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/44b5c8db-6c09-4527-b440-09d01bde3588%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:50am UTC](https://discuss.elastic.co/t/es-1-3-4-scrolling-never-ends/20578/11 "2017-07-06T00:50:25Z")

</div>