Scroll api issue

Elastic Stack Elasticsearch
1

Hello, I am trying to use the scroll API in Python, and i having an issue for it looping through my whole dataset.

I get about 100 results returned when there should be over 150k of them (I can see them in kibana)

attached is my code

res = helpers.scan(client = es, scroll = '2m', query = {
      "size": 10000,
        "query": {
          "match": {
            "type": {
              "query": "SSH-Telnet_Threats"
            }}}}, 
    index = "logstash-*")

# function to return hits from the elasticsearch query in res

def get_es_json(es_scan):
    for hits in es_scan:
        return hits

# iterate through results with defined number of results

def return_es_results(es_json_data, num_results):
    for i in range(num_results):
        data = get_es_json(es_json_data)
        print(data['_source']['geoip']['asn'])

return_es_results(res, 100)
2

also posted here https://stackoverflow.com/questions/56049910/elasticsearch-scroll-api-issue

3

I replied in stackoverflow.

In your call you asked for 100 results

return_es_results(res, 100)

So it's returning 100 results....
you may want to paginate?
if you use django there's some documentation here about pagination:
https://docs.djangoproject.com/en/2.2/topics/pagination/

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.