Hi there,
I'm trying to get a whole bunch of data out of elastic and into python to do some stuff with it.
It was going fine with the normal search api but it is limited to 10.000 and I need a lot more.
I am using the elasticsearch-dsl library for python which uses the scroll function of elastic. However I am getting a weird error when trying to get the data about which I can't find much information and I've got no clue how to fix it.
This is the code:
from elasticsearch_dsl import Search
search = Search(using=es).filter('range', **{'@timestamp': {'gte': 1526454000000 , 'lt': 1526489999999}})
search = search.source(['notifications.deviceId', 'notifications.locationCoordinate', '@timestamp'])
search.execute()
for d in search.scan():
print(d)
The print is just to test if it does anything, but later I have the stuff I wanna do with the data instead.
The print prints some data but then throws an error, anything else just immediately throws the error. Which is:
ScanError: Scroll request has only succeeded on 696 shards out of 1004.
I haven't been working with elastic that long and I'm not sure what this means. I couldn't find much about this particular error online. So hopefully someone has a clue what it is and how to fix it.