# Scan and Scroll Problem **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2012, 11:16am UTC](https://discuss.elastic.co/t/scan-and-scroll-problem/9627 "2012-11-08T11:16:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![smhdiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smhdiu/32/1808_2.png) [@smhdiu](https://discuss.elastic.co/u/smhdiu)\ **Post date:** [November 8, 2012, 11:16am UTC](https://discuss.elastic.co/t/scan-and-scroll-problem/9627/1 "2012-11-08T11:16:42Z")
Hello, here is my start scroll method. public SearchResponse startScroll(String searchText, long journalId, String indexName) { ``` QueryBuilder queryBuilder = QueryBuilders.queryString(searchText) .field("email").field("firstName") .field("lastName").field("authorOfAbstracts") .defaultOperator(QueryStringQueryBuilder.Operator.AND); SearchRequestBuilder searchRequestBuilder = ``` client.prepareSearch(indexName); searchRequestBuilder.setSearchType(SearchType.SCAN); searchRequestBuilder.setQuery(queryBuilder); searchRequestBuilder.setSize(BATCH\_SIZE); searchRequestBuilder.setExplain(false); searchRequestBuilder.setNoFields(); searchRequestBuilder.setVersion(true); searchRequestBuilder.setScroll(TimeValue.timeValueMillis(SCROLL\_TIME\_IN\_MILIS)); ``` return searchRequestBuilder.execute().actionGet(); } ``` and startIteration and returning the List of searchResponse ``` public List startIterationOnScroll(String scrollId){ List responseList = new ArrayList(); SearchResponse batchSearchResponse = null; do { batchSearchResponse = ``` client.prepareSearchScroll(scrollId).setScroll(TimeValue.timeValueMillis(SCROLL\_TIME\_IN\_MILIS)).execute().actionGet(); if(batchSearchResponse.getHits().getHits().length \> 0) responseList.add(batchSearchResponse); [//System.out.println](https://System.out.println)("hit : " + batchSearchResponse.getHits().getHits().length); scrollId = batchSearchResponse.getScrollId(); } while(batchSearchResponse.getHits().getHits().length \> 0); ``` return responseList; } ``` now once i have the list of responses i am converting all the results to java object as below code using jackson as david's code. but i am getting hit.getSourceAsString() null all the time in interation. i do have data in SearchResponse as below. Total Hits : 2 contact size : { "\_scroll\_id" : "c2NhbjswOzE7dG90YWxfaGl0czoyOw==", "took" : 21, "timed\_out" : false, "\_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 2, "max\_score" : 0.0, "hits" : [ { "\_index" : "contactsearch", "\_type" : "2037", "\_id" : "9959080154290929", "\_version" : 1, "\_score" : 0.0 }, { "\_index" : "contactsearch", "\_type" : "9001", "\_id" : "9959080154290929", "\_version" : 1, "\_score" : 0.0 } ] } } public List getContactList(List searchResponses) throws IOException { List contactList = new ArrayList(); String source =""; for(SearchResponse searchResponse : searchResponses) { for(SearchHit hit : searchResponse.getHits()){ source = hit.getSourceAsString(); if(source != null) { Contact contact = fromJSON(source); contactList.add(contact); } } System.out.println("contact size : " + searchResponse); } ``` return contactList; } private Contact fromJSON(String json) throws IOException { ObjectMapper mapper = new ObjectMapper(); return mapper.readValue(json.getBytes(), Contact.class); } ``` please please suggest some help Thanks -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [November 9, 2012, 6:34pm UTC](https://discuss.elastic.co/t/scan-and-scroll-problem/9627/2 "2012-11-09T18:34:05Z")
This line prevents elasticsearch from returning any fields in the response except id and type: searchRequestBuilder.setNoFields(); On Thursday, November 8, 2012 6:16:42 AM UTC-5, mohsin husen wrote: > Hello, > > here is my start scroll method. > > public SearchResponse startScroll(String searchText, long journalId, > String indexName) { > > ``` > QueryBuilder queryBuilder = QueryBuilders.queryString(searchText) > .field("email").field("firstName") > .field("lastName").field("authorOfAbstracts") > .defaultOperator(QueryStringQueryBuilder.Operator.AND); > > SearchRequestBuilder searchRequestBuilder = > > ``` > > client.prepareSearch(indexName); > searchRequestBuilder.setSearchType(SearchType.SCAN); > searchRequestBuilder.setQuery(queryBuilder); > searchRequestBuilder.setSize(BATCH\_SIZE); > searchRequestBuilder.setExplain(false); > searchRequestBuilder.setNoFields(); > searchRequestBuilder.setVersion(true); > > searchRequestBuilder.setScroll(TimeValue.timeValueMillis(SCROLL\_TIME\_IN\_MILIS)); > > ``` > return searchRequestBuilder.execute().actionGet(); > } > > ``` > > and startIteration and returning the List of searchResponse > > ``` > public List startIterationOnScroll(String scrollId){ > List responseList = new > > ``` > > ArrayList(); > SearchResponse batchSearchResponse = null; > do { > batchSearchResponse = > client.prepareSearchScroll(scrollId).setScroll(TimeValue.timeValueMillis(SCROLL\_TIME\_IN\_MILIS)).execute().actionGet(); > if(batchSearchResponse.getHits().getHits().length \> 0) > responseList.add(batchSearchResponse); > [//System.out.println](https://System.out.println)("hit : " + > batchSearchResponse.getHits().getHits().length); > scrollId = batchSearchResponse.getScrollId(); > } while(batchSearchResponse.getHits().getHits().length \> 0); > > ``` > return responseList; > } > > ``` > > now once i have the list of responses i am converting all the results to > java object as below code using jackson as david's code. > but i am getting hit.getSourceAsString() null all the time in interation. > i do have data in SearchResponse as below. > > Total Hits : 2 > contact size : { > "\_scroll\_id" : "c2NhbjswOzE7dG90YWxfaGl0czoyOw==", > "took" : 21, > "timed\_out" : false, > "\_shards" : { > "total" : 5, > "successful" : 5, > "failed" : 0 > }, > "hits" : { > "total" : 2, > "max\_score" : 0.0, > "hits" : [ { > "\_index" : "contactsearch", > "\_type" : "2037", > "\_id" : "9959080154290929", > "\_version" : 1, > "\_score" : 0.0 > }, { > "\_index" : "contactsearch", > "\_type" : "9001", > "\_id" : "9959080154290929", > "\_version" : 1, > "\_score" : 0.0 > } ] > } > } > > public List getContactList(List searchResponses) > throws IOException { > List contactList = new ArrayList(); > String source =""; > for(SearchResponse searchResponse : searchResponses) > { > for(SearchHit hit : searchResponse.getHits()){ > source = hit.getSourceAsString(); > if(source != null) > { > Contact contact = fromJSON(source); > contactList.add(contact); > } > } > System.out.println("contact size : " + searchResponse); > } > > ``` > return contactList; > } > > private Contact fromJSON(String json) throws IOException { > ObjectMapper mapper = new ObjectMapper(); > return mapper.readValue(json.getBytes(), Contact.class); > } > > ``` > > please please suggest some help > > Thanks -- ---
**Author:** ![smhdiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smhdiu/32/1808_2.png) [@smhdiu](https://discuss.elastic.co/u/smhdiu)\ **Post date:** [November 12, 2012, 11:23am UTC](https://discuss.elastic.co/t/scan-and-scroll-problem/9627/3 "2012-11-12T11:23:53Z")
stupid meeeeeeee ! Thanks Igor 🙂 On Friday, 9 November 2012 18:34:06 UTC, Igor Motov wrote: > This line prevents elasticsearch from returning any fields in the > response except id and type: searchRequestBuilder.setNoFields(); > > On Thursday, November 8, 2012 6:16:42 AM UTC-5, mohsin husen wrote: > > > Hello, > > > > here is my start scroll method. > > > > public SearchResponse startScroll(String searchText, long journalId, > > String indexName) { > > > > ``` > > QueryBuilder queryBuilder = QueryBuilders.queryString(searchText) > > .field("email").field("firstName") > > .field("lastName").field("authorOfAbstracts") > > .defaultOperator(QueryStringQueryBuilder.Operator.AND); > > > > SearchRequestBuilder searchRequestBuilder = > > > > ``` > > > > client.prepareSearch(indexName); > > searchRequestBuilder.setSearchType(SearchType.SCAN); > > searchRequestBuilder.setQuery(queryBuilder); > > searchRequestBuilder.setSize(BATCH\_SIZE); > > searchRequestBuilder.setExplain(false); > > searchRequestBuilder.setNoFields(); > > searchRequestBuilder.setVersion(true); > > > > searchRequestBuilder.setScroll(TimeValue.timeValueMillis(SCROLL\_TIME\_IN\_MILIS)); > > > > ``` > > return searchRequestBuilder.execute().actionGet(); > > } > > > > ``` > > > > and startIteration and returning the List of searchResponse > > > > ``` > > public List startIterationOnScroll(String scrollId){ > > List responseList = new > > > > ``` > > > > ArrayList(); > > SearchResponse batchSearchResponse = null; > > do { > > batchSearchResponse = > > client.prepareSearchScroll(scrollId).setScroll(TimeValue.timeValueMillis(SCROLL\_TIME\_IN\_MILIS)).execute().actionGet(); > > if(batchSearchResponse.getHits().getHits().length \> 0) > > responseList.add(batchSearchResponse); > > [//System.out.println](https://System.out.println)("hit : " + > > batchSearchResponse.getHits().getHits().length); > > scrollId = batchSearchResponse.getScrollId(); > > } while(batchSearchResponse.getHits().getHits().length \> 0); > > > > ``` > > return responseList; > > } > > > > ``` > > > > now once i have the list of responses i am converting all the results to > > java object as below code using jackson as david's code. > > but i am getting hit.getSourceAsString() null all the time in interation. > > i do have data in SearchResponse as below. > > > > Total Hits : 2 > > contact size : { > > "\_scroll\_id" : "c2NhbjswOzE7dG90YWxfaGl0czoyOw==", > > "took" : 21, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 5, > > "successful" : 5, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 2, > > "max\_score" : 0.0, > > "hits" : [ { > > "\_index" : "contactsearch", > > "\_type" : "2037", > > "\_id" : "9959080154290929", > > "\_version" : 1, > > "\_score" : 0.0 > > }, { > > "\_index" : "contactsearch", > > "\_type" : "9001", > > "\_id" : "9959080154290929", > > "\_version" : 1, > > "\_score" : 0.0 > > } ] > > } > > } > > > > public List getContactList(List searchResponses) > > throws IOException { > > List contactList = new ArrayList(); > > String source =""; > > for(SearchResponse searchResponse : searchResponses) > > { > > for(SearchHit hit : searchResponse.getHits()){ > > source = hit.getSourceAsString(); > > if(source != null) > > { > > Contact contact = fromJSON(source); > > contactList.add(contact); > > } > > } > > System.out.println("contact size : " + searchResponse); > > } > > > > ``` > > return contactList; > > } > > > > private Contact fromJSON(String json) throws IOException { > > ObjectMapper mapper = new ObjectMapper(); > > return mapper.readValue(json.getBytes(), Contact.class); > > } > > > > ``` > > > > please please suggest some help > > > > Thanks -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/scan-and-scroll-problem/9627/4 "2017-07-06T03:04:46Z")