# Number of Docs Returned

**URL:** <https://discuss.elastic.co/t/number-of-docs-returned/5328>\
**Category:** Elasticsearch\
**Created:** [September 7, 2011, 6:16pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328 "2011-09-07T18:16:34Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adam\_Estrada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_estrada/32/2704_2.png) [@Adam\_Estrada](https://discuss.elastic.co/u/Adam_Estrada)\
**Post date:** [September 7, 2011, 6:16pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/1 "2011-09-07T18:16:34Z")

</div>

All,

I am doing a search using the pyes geodistancefilter and I am only  
getting the top 10 results at a time. That is by design, right? I want  
to be able to retrieve all my the docs at a time and write them to a  
file. There are about 48k of them. Am I missing anything from the  
following? You can see that I am getting the total which is iterating  
by the number of docs found but its writing the same records over and  
over again. How can I get the next set of records in the series?

from pyes import ES  
from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
GeoPolygonFilter, FilteredQuery, MatchAllQuery

conn = ES('localhost:9200')

def make\_corpus():  
try:  
gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
"400km")  
q = FilteredQuery(MatchAllQuery(), gq)  
rs = conn.search(query=q, indices=["getdata"])  
hits = rs['hits']  
total = hits['total']

```
    file = open('c:\\Temp\\corpus.txt', 'wb')

    for x in xrange(total):
        recs = hits['hits']
        for x in recs:
            source = x['_source']
            str = source['properties']['translated']
            text = str.split()
            final = ' '.join(text)
            file.writelines(final + '\n')
    file.close()

except Exception as err:
    print err

```

Thanks,  
Adam

---

<div class="post-metadata">

**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\
**Post date:** [September 7, 2011, 6:35pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/2 "2011-09-07T18:35:15Z")

</div>

You need to page though the results using from and size. Defaults are from  
= 0, and size = 10. So by default you get the first 10 results, do get the  
next 10 results you will set from = 10 and leave size alone. Never used  
pyes before but just glancing at it you will need to get a search object and  
then set the from parameter on that. Don't return 48k results in a single  
call.

```
   gq = GeoDistanceFilter("geometry.coordinates", [72, 31],

```

"400km")  
q = FilteredQuery(MatchAllQuery(), gq)  
qs = q.search()

rs = conn.search(query=qs, indices=["getdata"])

Thanks,  
Matt Weber

On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada [estrada.adam@gmail.com](mailto:estrada.adam@gmail.com)wrote:

> All,
> 
> I am doing a search using the pyes geodistancefilter and I am only  
> getting the top 10 results at a time. That is by design, right? I want  
> to be able to retrieve all my the docs at a time and write them to a  
> file. There are about 48k of them. Am I missing anything from the  
> following? You can see that I am getting the total which is iterating  
> by the number of docs found but its writing the same records over and  
> over again. How can I get the next set of records in the series?
> 
> from pyes import ES  
> from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> GeoPolygonFilter, FilteredQuery, MatchAllQuery
> 
> conn = ES('localhost:9200')
> 
> def make\_corpus():  
> try:  
> gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> "400km")  
> q = FilteredQuery(MatchAllQuery(), gq)  
> rs = conn.search(query=q, indices=["getdata"])  
> hits = rs['hits']  
> total = hits['total']
> 
> ```
> file = open('c:\\Temp\\corpus.txt', 'wb')
> 
> for x in xrange(total):
> recs = hits['hits']
> for x in recs:
> source = x['_source']
> str = source['properties']['translated']
> text = str.split()
> final = ' '.join(text)
> file.writelines(final + '\n')
> file.close()
> 
> ```
> 
> except Exception as err:  
> print err
> 
> Thanks,  
> Adam

---

<div class="post-metadata">

**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\
**Post date:** [September 7, 2011, 6:39pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/3 "2011-09-07T18:39:10Z")

</div>

Sent too soon.. after your first search you will know the total hit count,  
divide this by size and then start executing more searches:

totalreq = total / 10  
for i in range(1, totalreq):  
qs.start = i \* 10  
rs = conn.search(query=qs, indices=["getdata"])

Something like that should work.

Thanks,  
Matt Weber

On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [matt@mattweber.org](mailto:matt@mattweber.org) wrote:

> You need to page though the results using from and size. Defaults are from  
> = 0, and size = 10. So by default you get the first 10 results, do get the  
> next 10 results you will set from = 10 and leave size alone. Never used  
> pyes before but just glancing at it you will need to get a search object and  
> then set the from parameter on that. Don't return 48k results in a single  
> call.
> 
> ```
> gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> 
> ```
> 
> "400km")  
> q = FilteredQuery(MatchAllQuery(), gq)  
> qs = q.search()
> 
> rs = conn.search(query=qs, indices=["getdata"])
> 
> Thanks,  
> Matt Weber
> 
> On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada [estrada.adam@gmail.com](mailto:estrada.adam@gmail.com)wrote:
> 
> > All,
> > 
> > I am doing a search using the pyes geodistancefilter and I am only  
> > getting the top 10 results at a time. That is by design, right? I want  
> > to be able to retrieve all my the docs at a time and write them to a  
> > file. There are about 48k of them. Am I missing anything from the  
> > following? You can see that I am getting the total which is iterating  
> > by the number of docs found but its writing the same records over and  
> > over again. How can I get the next set of records in the series?
> > 
> > from pyes import ES  
> > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> > 
> > conn = ES('localhost:9200')
> > 
> > def make\_corpus():  
> > try:  
> > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > "400km")  
> > q = FilteredQuery(MatchAllQuery(), gq)  
> > rs = conn.search(query=q, indices=["getdata"])  
> > hits = rs['hits']  
> > total = hits['total']
> > 
> > ```
> > file = open('c:\\Temp\\corpus.txt', 'wb')
> > 
> > for x in xrange(total):
> > recs = hits['hits']
> > for x in recs:
> > source = x['_source']
> > str = source['properties']['translated']
> > text = str.split()
> > final = ' '.join(text)
> > file.writelines(final + '\n')
> > file.close()
> > 
> > ```
> > 
> > except Exception as err:  
> > print err
> > 
> > Thanks,  
> > Adam

---

<div class="post-metadata">

**Author:** ![Adam\_Estrada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_estrada/32/2704_2.png) [@Adam\_Estrada](https://discuss.elastic.co/u/Adam_Estrada)\
**Post date:** [September 7, 2011, 7:56pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/4 "2011-09-07T19:56:18Z")

</div>

w00t! Thanks a lot! Corpus time!!! Actually, I need to do this to work  
with Mahout so if anyone has any tips for extracting topics out of  
there, please let me know...

A

def make\_corpus():  
try:  
gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
"400km")  
q = FilteredQuery(MatchAllQuery(), gq)  
qs = q.search()  
rs = conn.search(query=q, indices=["getdata"])  
total = rs['hits']['total']

```
    file = open('c:\\Temp\\corpus.csv', 'wb')
    print total
    totalReq = total / 10
    file.write('id' + ',' + 'text' + '\n')
    for i in xrange(1, totalReq):
        qs.start = i * 10
        rslt = conn.search(query=qs, indices=["getdata"])
        hits = rslt['hits']['hits']

        for x in hits:
            id = x['_id']
            source = x['_source']
            str = source['properties']['translated']
            text = str.split()
            final = ' '.join(text).strip()
            #file.writelines(final + '\n')
            file.writelines(id + ',' + final + '\n')
    file.close()

except Exception as err:
    print err

```

On Sep 7, 2:39 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:

> Sent too soon.. after your first search you will know the total hit count,  
> divide this by size and then start executing more searches:
> 
> totalreq = total / 10  
> for i in range(1, totalreq):  
> qs.start = i \* 10  
> rs = conn.search(query=qs, indices=["getdata"])
> 
> Something like that should work.
> 
> Thanks,  
> Matt Weber
> 
> On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> 
> > You need to page though the results using from and size. Defaults are from  
> > = 0, and size = 10. So by default you get the first 10 results, do get the  
> > next 10 results you will set from = 10 and leave size alone. Never used  
> > pyes before but just glancing at it you will need to get a search object and  
> > then set the from parameter on that. Don't return 48k results in a single  
> > call.
> 
> > ```
> > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> > 
> > ```
> > 
> > "400km")  
> > q = FilteredQuery(MatchAllQuery(), gq)  
> > qs = q.search()
> 
> > rs = conn.search(query=qs, indices=["getdata"])
> 
> > Thanks,  
> > Matt Weber
> 
> > On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)wrote:
> 
> > > All,
> 
> > > I am doing a search using the pyes geodistancefilter and I am only  
> > > getting the top 10 results at a time. That is by design, right? I want  
> > > to be able to retrieve all my the docs at a time and write them to a  
> > > file. There are about 48k of them. Am I missing anything from the  
> > > following? You can see that I am getting the total which is iterating  
> > > by the number of docs found but its writing the same records over and  
> > > over again. How can I get the next set of records in the series?
> 
> > > from pyes import ES  
> > > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> 
> > > conn = ES('localhost:9200')
> 
> > > def make\_corpus():  
> > > try:  
> > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > "400km")  
> > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > rs = conn.search(query=q, indices=["getdata"])  
> > > hits = rs['hits']  
> > > total = hits['total']
> 
> > > ```
> > > file = open('c:\\Temp\\corpus.txt', 'wb')
> > > 
> > > ```
> 
> > > ```
> > > for x in xrange(total):
> > > recs = hits['hits']
> > > for x in recs:
> > > source = x['_source']
> > > str = source['properties']['translated']
> > > text = str.split()
> > > final = ' '.join(text)
> > > file.writelines(final + '\n')
> > > file.close()
> > > 
> > > ```
> 
> > > except Exception as err:  
> > > print err
> 
> > > Thanks,  
> > > Adam

---

<div class="post-metadata">

**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\
**Post date:** [September 7, 2011, 9:10pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/5 "2011-09-07T21:10:52Z")

</div>

Actually with that code you are skipping the first 10 results, make sure you  
process the hits for the first conn.search call.

Thanks,  
Matt Weber

On Wed, Sep 7, 2011 at 12:56 PM, Adam Estrada [estrada.adam@gmail.com](mailto:estrada.adam@gmail.com)wrote:

> w00t! Thanks a lot! Corpus time!!! Actually, I need to do this to work  
> with Mahout so if anyone has any tips for extracting topics out of  
> there, please let me know...
> 
> A
> 
> def make\_corpus():  
> try:  
> gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> "400km")  
> q = FilteredQuery(MatchAllQuery(), gq)  
> qs = q.search()  
> rs = conn.search(query=q, indices=["getdata"])  
> total = rs['hits']['total']
> 
> ```
> file = open('c:\\Temp\\corpus.csv', 'wb')
> print total
> totalReq = total / 10
> file.write('id' + ',' + 'text' + '\n')
> for i in xrange(1, totalReq):
> qs.start = i * 10
> rslt = conn.search(query=qs, indices=["getdata"])
> hits = rslt['hits']['hits']
> 
> for x in hits:
> id = x['_id']
> source = x['_source']
> str = source['properties']['translated']
> text = str.split()
> final = ' '.join(text).strip()
> #file.writelines(final + '\n')
> file.writelines(id + ',' + final + '\n')
> file.close()
> 
> ```
> 
> except Exception as err:  
> print err
> 
> On Sep 7, 2:39 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> 
> > Sent too soon.. after your first search you will know the total hit  
> > count,  
> > divide this by size and then start executing more searches:
> > 
> > totalreq = total / 10  
> > for i in range(1, totalreq):  
> > qs.start = i \* 10  
> > rs = conn.search(query=qs, indices=["getdata"])
> > 
> > Something like that should work.
> > 
> > Thanks,  
> > Matt Weber
> > 
> > On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > 
> > > You need to page though the results using from and size. Defaults are  
> > > from  
> > > = 0, and size = 10. So by default you get the first 10 results, do get  
> > > the  
> > > next 10 results you will set from = 10 and leave size alone. Never  
> > > used  
> > > pyes before but just glancing at it you will need to get a search  
> > > object and  
> > > then set the from parameter on that. Don't return 48k results in a  
> > > single  
> > > call.
> > 
> > > ```
> > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> > > 
> > > ```
> > > 
> > > "400km")  
> > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > qs = q.search()
> > 
> > > rs = conn.search(query=qs, indices=["getdata"])
> > 
> > > Thanks,  
> > > Matt Weber
> > 
> > > On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada \<[estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)  
> > > wrote:
> > 
> > > > All,
> > 
> > > > I am doing a search using the pyes geodistancefilter and I am only  
> > > > getting the top 10 results at a time. That is by design, right? I want  
> > > > to be able to retrieve all my the docs at a time and write them to a  
> > > > file. There are about 48k of them. Am I missing anything from the  
> > > > following? You can see that I am getting the total which is iterating  
> > > > by the number of docs found but its writing the same records over and  
> > > > over again. How can I get the next set of records in the series?
> > 
> > > > from pyes import ES  
> > > > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > > > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> > 
> > > > conn = ES('localhost:9200')
> > 
> > > > def make\_corpus():  
> > > > try:  
> > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > "400km")  
> > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > hits = rs['hits']  
> > > > total = hits['total']
> > 
> > > > ```
> > > > file = open('c:\\Temp\\corpus.txt', 'wb')
> > > > 
> > > > ```
> > 
> > > > ```
> > > > for x in xrange(total):
> > > > recs = hits['hits']
> > > > for x in recs:
> > > > source = x['_source']
> > > > str = source['properties']['translated']
> > > > text = str.split()
> > > > final = ' '.join(text)
> > > > file.writelines(final + '\n')
> > > > file.close()
> > > > 
> > > > ```
> > 
> > > > except Exception as err:  
> > > > print err
> > 
> > > > Thanks,  
> > > > Adam

---

<div class="post-metadata">

**Author:** ![Adam\_Estrada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_estrada/32/2704_2.png) [@Adam\_Estrada](https://discuss.elastic.co/u/Adam_Estrada)\
**Post date:** [September 8, 2011, 3:31pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/6 "2011-09-08T15:31:18Z")

</div>

Good catch! How the heck do I increase the size of the fetch? I have  
increased it to 1000 but it still fetches 10 at a time.

adam

On Sep 7, 5:10 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:

> Actually with that code you are skipping the first 10 results, make sure you  
> process the hits for the first conn.search call.
> 
> Thanks,  
> Matt Weber
> 
> On Wed, Sep 7, 2011 at 12:56 PM, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)wrote:
> 
> > w00t! Thanks a lot! Corpus time!!! Actually, I need to do this to work  
> > with Mahout so if anyone has any tips for extracting topics out of  
> > there, please let me know...
> 
> > A
> 
> > def make\_corpus():  
> > try:  
> > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > "400km")  
> > q = FilteredQuery(MatchAllQuery(), gq)  
> > qs = q.search()  
> > rs = conn.search(query=q, indices=["getdata"])  
> > total = rs['hits']['total']
> 
> > ```
> > file = open('c:\\Temp\\corpus.csv', 'wb')
> > print total
> > totalReq = total / 10
> > file.write('id' + ',' + 'text' + '\n')
> > for i in xrange(1, totalReq):
> > qs.start = i * 10
> > rslt = conn.search(query=qs, indices=["getdata"])
> > hits = rslt['hits']['hits']
> > 
> > ```
> 
> > ```
> > for x in hits:
> > id = x['_id']
> > source = x['_source']
> > str = source['properties']['translated']
> > text = str.split()
> > final = ' '.join(text).strip()
> > #file.writelines(final + '\n')
> > file.writelines(id + ',' + final + '\n')
> > file.close()
> > 
> > ```
> 
> > except Exception as err:  
> > print err
> 
> > On Sep 7, 2:39 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > 
> > > Sent too soon.. after your first search you will know the total hit  
> > > count,  
> > > divide this by size and then start executing more searches:
> 
> > > totalreq = total / 10  
> > > for i in range(1, totalreq):  
> > > qs.start = i \* 10  
> > > rs = conn.search(query=qs, indices=["getdata"])
> 
> > > Something like that should work.
> 
> > > Thanks,  
> > > Matt Weber
> 
> > > On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > > 
> > > > You need to page though the results using from and size. Defaults are  
> > > > from  
> > > > = 0, and size = 10. So by default you get the first 10 results, do get  
> > > > the  
> > > > next 10 results you will set from = 10 and leave size alone. Never  
> > > > used  
> > > > pyes before but just glancing at it you will need to get a search  
> > > > object and  
> > > > then set the from parameter on that. Don't return 48k results in a  
> > > > single  
> > > > call.
> 
> > > > ```
> > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> > > > 
> > > > ```
> > > > 
> > > > "400km")  
> > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > qs = q.search()
> 
> > > > rs = conn.search(query=qs, indices=["getdata"])
> 
> > > > Thanks,  
> > > > Matt Weber
> 
> > > > On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada \<[estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)  
> > > > wrote:
> 
> > > > > All,
> 
> > > > > I am doing a search using the pyes geodistancefilter and I am only  
> > > > > getting the top 10 results at a time. That is by design, right? I want  
> > > > > to be able to retrieve all my the docs at a time and write them to a  
> > > > > file. There are about 48k of them. Am I missing anything from the  
> > > > > following? You can see that I am getting the total which is iterating  
> > > > > by the number of docs found but its writing the same records over and  
> > > > > over again. How can I get the next set of records in the series?
> 
> > > > > from pyes import ES  
> > > > > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > > > > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> 
> > > > > conn = ES('localhost:9200')
> 
> > > > > def make\_corpus():  
> > > > > try:  
> > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > > "400km")  
> > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > > hits = rs['hits']  
> > > > > total = hits['total']
> 
> > > > > ```
> > > > > file = open('c:\\Temp\\corpus.txt', 'wb')
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > for x in xrange(total):
> > > > > recs = hits['hits']
> > > > > for x in recs:
> > > > > source = x['_source']
> > > > > str = source['properties']['translated']
> > > > > text = str.split()
> > > > > final = ' '.join(text)
> > > > > file.writelines(final + '\n')
> > > > > file.close()
> > > > > 
> > > > > ```
> 
> > > > > except Exception as err:  
> > > > > print err
> 
> > > > > Thanks,  
> > > > > Adam

---

<div class="post-metadata">

**Author:** ![Adam\_Estrada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adam_estrada/32/2704_2.png) [@Adam\_Estrada](https://discuss.elastic.co/u/Adam_Estrada)\
**Post date:** [September 8, 2011, 3:59pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/7 "2011-09-08T15:59:54Z")

</div>

I dug through the sources and found that you can set the size directly  
in the search

rs = conn.search(query=q, indices=["getdata"], size=1000)

Cool!  
A

On Sep 8, 11:31 am, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com) wrote:

> Good catch! How the heck do I increase the size of the fetch? I have  
> increased it to 1000 but it still fetches 10 at a time.
> 
> adam
> 
> On Sep 7, 5:10 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> 
> > Actually with that code you are skipping the first 10 results, make sure you  
> > process the hits for the first conn.search call.
> 
> > Thanks,  
> > Matt Weber
> 
> > On Wed, Sep 7, 2011 at 12:56 PM, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)wrote:
> 
> > > w00t! Thanks a lot! Corpus time!!! Actually, I need to do this to work  
> > > with Mahout so if anyone has any tips for extracting topics out of  
> > > there, please let me know...
> 
> > > A
> 
> > > def make\_corpus():  
> > > try:  
> > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > "400km")  
> > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > qs = q.search()  
> > > rs = conn.search(query=q, indices=["getdata"])  
> > > total = rs['hits']['total']
> 
> > > ```
> > > file = open('c:\\Temp\\corpus.csv', 'wb')
> > > print total
> > > totalReq = total / 10
> > > file.write('id' + ',' + 'text' + '\n')
> > > for i in xrange(1, totalReq):
> > > qs.start = i * 10
> > > rslt = conn.search(query=qs, indices=["getdata"])
> > > hits = rslt['hits']['hits']
> > > 
> > > ```
> 
> > > ```
> > > for x in hits:
> > > id = x['_id']
> > > source = x['_source']
> > > str = source['properties']['translated']
> > > text = str.split()
> > > final = ' '.join(text).strip()
> > > #file.writelines(final + '\n')
> > > file.writelines(id + ',' + final + '\n')
> > > file.close()
> > > 
> > > ```
> 
> > > except Exception as err:  
> > > print err
> 
> > > On Sep 7, 2:39 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > > 
> > > > Sent too soon.. after your first search you will know the total hit  
> > > > count,  
> > > > divide this by size and then start executing more searches:
> 
> > > > totalreq = total / 10  
> > > > for i in range(1, totalreq):  
> > > > qs.start = i \* 10  
> > > > rs = conn.search(query=qs, indices=["getdata"])
> 
> > > > Something like that should work.
> 
> > > > Thanks,  
> > > > Matt Weber
> 
> > > > On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > > > 
> > > > > You need to page though the results using from and size. Defaults are  
> > > > > from  
> > > > > = 0, and size = 10. So by default you get the first 10 results, do get  
> > > > > the  
> > > > > next 10 results you will set from = 10 and leave size alone. Never  
> > > > > used  
> > > > > pyes before but just glancing at it you will need to get a search  
> > > > > object and  
> > > > > then set the from parameter on that. Don't return 48k results in a  
> > > > > single  
> > > > > call.
> 
> > > > > ```
> > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> > > > > 
> > > > > ```
> > > > > 
> > > > > "400km")  
> > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > qs = q.search()
> 
> > > > > rs = conn.search(query=qs, indices=["getdata"])
> 
> > > > > Thanks,  
> > > > > Matt Weber
> 
> > > > > On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada \<[estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)  
> > > > > wrote:
> 
> > > > > > All,
> 
> > > > > > I am doing a search using the pyes geodistancefilter and I am only  
> > > > > > getting the top 10 results at a time. That is by design, right? I want  
> > > > > > to be able to retrieve all my the docs at a time and write them to a  
> > > > > > file. There are about 48k of them. Am I missing anything from the  
> > > > > > following? You can see that I am getting the total which is iterating  
> > > > > > by the number of docs found but its writing the same records over and  
> > > > > > over again. How can I get the next set of records in the series?
> 
> > > > > > from pyes import ES  
> > > > > > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > > > > > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> 
> > > > > > conn = ES('localhost:9200')
> 
> > > > > > def make\_corpus():  
> > > > > > try:  
> > > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > > > "400km")  
> > > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > > > hits = rs['hits']  
> > > > > > total = hits['total']
> 
> > > > > > ```
> > > > > > file = open('c:\\Temp\\corpus.txt', 'wb')
> > > > > > 
> > > > > > ```
> 
> > > > > > ```
> > > > > > for x in xrange(total):
> > > > > > recs = hits['hits']
> > > > > > for x in recs:
> > > > > > source = x['_source']
> > > > > > str = source['properties']['translated']
> > > > > > text = str.split()
> > > > > > final = ' '.join(text)
> > > > > > file.writelines(final + '\n')
> > > > > > file.close()
> > > > > > 
> > > > > > ```
> 
> > > > > > except Exception as err:  
> > > > > > print err
> 
> > > > > > Thanks,  
> > > > > > Adam

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [September 8, 2011, 5:49pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/8 "2011-09-08T17:49:05Z")

</div>

By the way, a more performant way is to use scan search type here, not sure  
how its exposed in pyes.  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html).

On Thu, Sep 8, 2011 at 6:59 PM, Adam Estrada [estrada.adam@gmail.com](mailto:estrada.adam@gmail.com) wrote:

> I dug through the sources and found that you can set the size directly  
> in the search
> 
> rs = conn.search(query=q, indices=["getdata"], size=1000)
> 
> Cool!  
> A
> 
> On Sep 8, 11:31 am, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com) wrote:
> 
> > Good catch! How the heck do I increase the size of the fetch? I have  
> > increased it to 1000 but it still fetches 10 at a time.
> > 
> > adam
> > 
> > On Sep 7, 5:10 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > 
> > > Actually with that code you are skipping the first 10 results, make  
> > > sure you  
> > > process the hits for the first conn.search call.
> > 
> > > Thanks,  
> > > Matt Weber
> > 
> > > On Wed, Sep 7, 2011 at 12:56 PM, Adam Estrada \<[estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)  
> > > wrote:
> > 
> > > > w00t! Thanks a lot! Corpus time!!! Actually, I need to do this to  
> > > > work  
> > > > with Mahout so if anyone has any tips for extracting topics out of  
> > > > there, please let me know...
> > 
> > > > A
> > 
> > > > def make\_corpus():  
> > > > try:  
> > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > "400km")  
> > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > qs = q.search()  
> > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > total = rs['hits']['total']
> > 
> > > > ```
> > > > file = open('c:\\Temp\\corpus.csv', 'wb')
> > > > print total
> > > > totalReq = total / 10
> > > > file.write('id' + ',' + 'text' + '\n')
> > > > for i in xrange(1, totalReq):
> > > > qs.start = i * 10
> > > > rslt = conn.search(query=qs, indices=["getdata"])
> > > > hits = rslt['hits']['hits']
> > > > 
> > > > ```
> > 
> > > > ```
> > > > for x in hits:
> > > > id = x['_id']
> > > > source = x['_source']
> > > > str = source['properties']['translated']
> > > > text = str.split()
> > > > final = ' '.join(text).strip()
> > > > #file.writelines(final + '\n')
> > > > file.writelines(id + ',' + final + '\n')
> > > > file.close()
> > > > 
> > > > ```
> > 
> > > > except Exception as err:  
> > > > print err
> > 
> > > > On Sep 7, 2:39 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > > > 
> > > > > Sent too soon.. after your first search you will know the total hit  
> > > > > count,  
> > > > > divide this by size and then start executing more searches:
> > 
> > > > > totalreq = total / 10  
> > > > > for i in range(1, totalreq):  
> > > > > qs.start = i \* 10  
> > > > > rs = conn.search(query=qs, indices=["getdata"])
> > 
> > > > > Something like that should work.
> > 
> > > > > Thanks,  
> > > > > Matt Weber
> > 
> > > > > On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org)  
> > > > > wrote:
> > > > > 
> > > > > > You need to page though the results using from and size.  
> > > > > > Defaults are  
> > > > > > from  
> > > > > > = 0, and size = 10. So by default you get the first 10 results,  
> > > > > > do get  
> > > > > > the  
> > > > > > next 10 results you will set from = 10 and leave size alone.  
> > > > > > Never  
> > > > > > used  
> > > > > > pyes before but just glancing at it you will need to get a search  
> > > > > > object and  
> > > > > > then set the from parameter on that. Don't return 48k results in  
> > > > > > a  
> > > > > > single  
> > > > > > call.
> > 
> > > > > > ```
> > > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > "400km")  
> > > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > > qs = q.search()
> > 
> > > > > > rs = conn.search(query=qs, indices=["getdata"])
> > 
> > > > > > Thanks,  
> > > > > > Matt Weber
> > 
> > > > > > On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada \<  
> > > > > > [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)  
> > > > > > wrote:
> > 
> > > > > > > All,
> > 
> > > > > > > I am doing a search using the pyes geodistancefilter and I am  
> > > > > > > only  
> > > > > > > getting the top 10 results at a time. That is by design, right?  
> > > > > > > I want  
> > > > > > > to be able to retrieve all my the docs at a time and write them  
> > > > > > > to a  
> > > > > > > file. There are about 48k of them. Am I missing anything from  
> > > > > > > the  
> > > > > > > following? You can see that I am getting the total which is  
> > > > > > > iterating  
> > > > > > > by the number of docs found but its writing the same records  
> > > > > > > over and  
> > > > > > > over again. How can I get the next set of records in the series?
> > 
> > > > > > > from pyes import ES  
> > > > > > > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > > > > > > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> > 
> > > > > > > conn = ES('localhost:9200')
> > 
> > > > > > > def make\_corpus():  
> > > > > > > try:  
> > > > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > > > > "400km")  
> > > > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > > > > hits = rs['hits']  
> > > > > > > total = hits['total']
> > 
> > > > > > > ```
> > > > > > > file = open('c:\\Temp\\corpus.txt', 'wb')
> > > > > > > 
> > > > > > > ```
> > 
> > > > > > > ```
> > > > > > > for x in xrange(total):
> > > > > > > recs = hits['hits']
> > > > > > > for x in recs:
> > > > > > > source = x['_source']
> > > > > > > str = source['properties']['translated']
> > > > > > > text = str.split()
> > > > > > > final = ' '.join(text)
> > > > > > > file.writelines(final + '\n')
> > > > > > > file.close()
> > > > > > > 
> > > > > > > ```
> > 
> > > > > > > except Exception as err:  
> > > > > > > print err
> > 
> > > > > > > Thanks,  
> > > > > > > Adam

---

<div class="post-metadata">

**Author:** ![Alberto\_Paro\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alberto_paro_2/32/1137_2.png) [@Alberto\_Paro\_2](https://discuss.elastic.co/u/Alberto_Paro_2)\
**Post date:** [September 11, 2011, 8:11pm UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/9 "2011-09-11T20:11:15Z")

</div>

To use scan you can simple pass the scan type:

```
    resultset = conn.search(Search(q, size=1000),
                         "get_data,
                      search_type='scan',
                      scroll="10m"):

for obj in resultset:
	str = obj['properties']['translated']
            ....

```

The resultset allows at the fist call to fire the scan and at the second to iterate on your results (hiding the scroll id iteration).

Hi,  
Alberto

Il giorno 08/set/2011, alle ore 19:49, Shay Banon ha scritto:

> By the way, a more performant way is to use scan search type here, not sure how its exposed in pyes. [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html).
> 
> On Thu, Sep 8, 2011 at 6:59 PM, Adam Estrada [estrada.adam@gmail.com](mailto:estrada.adam@gmail.com) wrote:  
> I dug through the sources and found that you can set the size directly  
> in the search
> 
> rs = conn.search(query=q, indices=["getdata"], size=1000)
> 
> Cool!  
> A
> 
> On Sep 8, 11:31 am, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com) wrote:
> 
> > Good catch! How the heck do I increase the size of the fetch? I have  
> > increased it to 1000 but it still fetches 10 at a time.
> > 
> > adam
> > 
> > On Sep 7, 5:10 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > 
> > > Actually with that code you are skipping the first 10 results, make sure you  
> > > process the hits for the first conn.search call.
> > 
> > > Thanks,  
> > > Matt Weber
> > 
> > > On Wed, Sep 7, 2011 at 12:56 PM, Adam Estrada [estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)wrote:
> > 
> > > > w00t! Thanks a lot! Corpus time!!! Actually, I need to do this to work  
> > > > with Mahout so if anyone has any tips for extracting topics out of  
> > > > there, please let me know...
> > 
> > > > A
> > 
> > > > def make\_corpus():  
> > > > try:  
> > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > "400km")  
> > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > qs = q.search()  
> > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > total = rs['hits']['total']
> > 
> > > > ```
> > > > file = open('c:\\Temp\\corpus.csv', 'wb')
> > > > print total
> > > > totalReq = total / 10
> > > > file.write('id' + ',' + 'text' + '\n')
> > > > for i in xrange(1, totalReq):
> > > > qs.start = i * 10
> > > > rslt = conn.search(query=qs, indices=["getdata"])
> > > > hits = rslt['hits']['hits']
> > > > 
> > > > ```
> > 
> > > > ```
> > > > for x in hits:
> > > > id = x['_id']
> > > > source = x['_source']
> > > > str = source['properties']['translated']
> > > > text = str.split()
> > > > final = ' '.join(text).strip()
> > > > #file.writelines(final + '\n')
> > > > file.writelines(id + ',' + final + '\n')
> > > > file.close()
> > > > 
> > > > ```
> > 
> > > > except Exception as err:  
> > > > print err
> > 
> > > > On Sep 7, 2:39 pm, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > > > 
> > > > > Sent too soon.. after your first search you will know the total hit  
> > > > > count,  
> > > > > divide this by size and then start executing more searches:
> > 
> > > > > totalreq = total / 10  
> > > > > for i in range(1, totalreq):  
> > > > > qs.start = i \* 10  
> > > > > rs = conn.search(query=qs, indices=["getdata"])
> > 
> > > > > Something like that should work.
> > 
> > > > > Thanks,  
> > > > > Matt Weber
> > 
> > > > > On Wed, Sep 7, 2011 at 11:35 AM, Matt Weber [m...@mattweber.org](mailto:m...@mattweber.org) wrote:
> > > > > 
> > > > > > You need to page though the results using from and size. Defaults are  
> > > > > > from  
> > > > > > = 0, and size = 10. So by default you get the first 10 results, do get  
> > > > > > the  
> > > > > > next 10 results you will set from = 10 and leave size alone. Never  
> > > > > > used  
> > > > > > pyes before but just glancing at it you will need to get a search  
> > > > > > object and  
> > > > > > then set the from parameter on that. Don't return 48k results in a  
> > > > > > single  
> > > > > > call.
> > 
> > > > > > ```
> > > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > "400km")  
> > > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > > qs = q.search()
> > 
> > > > > > rs = conn.search(query=qs, indices=["getdata"])
> > 
> > > > > > Thanks,  
> > > > > > Matt Weber
> > 
> > > > > > On Wed, Sep 7, 2011 at 11:16 AM, Adam Estrada \<[estrada.a...@gmail.com](mailto:estrada.a...@gmail.com)  
> > > > > > wrote:
> > 
> > > > > > > All,
> > 
> > > > > > > I am doing a search using the pyes geodistancefilter and I am only  
> > > > > > > getting the top 10 results at a time. That is by design, right? I want  
> > > > > > > to be able to retrieve all my the docs at a time and write them to a  
> > > > > > > file. There are about 48k of them. Am I missing anything from the  
> > > > > > > following? You can see that I am getting the total which is iterating  
> > > > > > > by the number of docs found but its writing the same records over and  
> > > > > > > over again. How can I get the next set of records in the series?
> > 
> > > > > > > from pyes import ES  
> > > > > > > from pyes import GeoBoundingBoxFilter, GeoDistanceFilter,  
> > > > > > > GeoPolygonFilter, FilteredQuery, MatchAllQuery
> > 
> > > > > > > conn = ES('localhost:9200')
> > 
> > > > > > > def make\_corpus():  
> > > > > > > try:  
> > > > > > > gq = GeoDistanceFilter("geometry.coordinates", [72, 31],  
> > > > > > > "400km")  
> > > > > > > q = FilteredQuery(MatchAllQuery(), gq)  
> > > > > > > rs = conn.search(query=q, indices=["getdata"])  
> > > > > > > hits = rs['hits']  
> > > > > > > total = hits['total']
> > 
> > > > > > > ```
> > > > > > > file = open('c:\\Temp\\corpus.txt', 'wb')
> > > > > > > 
> > > > > > > ```
> > 
> > > > > > > ```
> > > > > > > for x in xrange(total):
> > > > > > > recs = hits['hits']
> > > > > > > for x in recs:
> > > > > > > source = x['_source']
> > > > > > > str = source['properties']['translated']
> > > > > > > text = str.split()
> > > > > > > final = ' '.join(text)
> > > > > > > file.writelines(final + '\n')
> > > > > > > file.close()
> > > > > > > 
> > > > > > > ```
> > 
> > > > > > > except Exception as err:  
> > > > > > > print err
> > 
> > > > > > > Thanks,  
> > > > > > > Adam

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:54am UTC](https://discuss.elastic.co/t/number-of-docs-returned/5328/10 "2017-07-06T03:54:46Z")

</div>