In my project requirement, I need to fetch more than 10k documents. I used ElasticSearch scroll api with python to do that. Here is my sample code -
url = '/_search/scroll'
scroll_url='//_search?scroll=2m'
Query= {"query": {"bool": {"must": [{"match_all": { }},{ "range": { "@timestamp": { "gt": "now-24h", "lt": "now-1h", "time_zone": "-06:00" } } }],"must_not": [ ],"should": [ ]}},"from": 0,"size":10,"sort": [ ],"aggs": { }}
response=requests.post(scroll_url, json=query)
sid = response['_scroll_id']
hits=response['hits']
total=hits["total"]
while(total>0):
scroll = '2m'
scroll_query=json.dumps({"scroll" : scroll, "scroll_id" : sid })
response1=rquests.post(url,data=scroll_query)
# Update the scroll ID
sid = response1['_scroll_id']
# Get the number of results that we returned in the last scroll
hits=response1['hits']
total=len(response1['hits']['hits'])
for each in hits['hits']:
#iterate through my document
Scroll work perfect the way I wanted to, but later I was informed that because of this scroll elasticsearch schema got corrupted and it recreated the indexes.
Is it true that scroll modify the ES structure or something wrong with my code. Please let me know.