Hi I see some data lose while using scroll. I am using the following inside python script.
Without using scroll it showing the correct number of documents but with scroll I am getting less number of documents.
Any idea about this ?
res = es.search(index=idx,scroll='1m',body=jobqry,timeout='1200s')
sid = res['_scroll_id']
scroll_size = res['hits']['total']
#print(scroll_size)
file_name = new_dir+"/"+clsname+".txt"
time.sleep(2)
cmd_file = "touch %s" %(file_name)
os.system(cmd_file)
while (scroll_size > 0):
res=es.scroll(scroll_id = sid, scroll ='1m')
sid = res['_scroll_id']
scroll_size = len(res['hits']['hits'])
for i in range(0,scroll_size-1):
print(res['hits']['hits'][i]['_source']['id'])