Elasticsearch 2.4.4
Hi,
I have a script where I'm creating trying to create a large number of documents in an index that already exist, testing something before it goes in production. I have a scroll size of 50k, number of documents being written is 3 million though this will need to scale to hundreds of millions.
Most of the 50k scroll are returning 409 document already exists exception for every document in the scroll. But sometimes its returning less than it should, sometimes less than half. And I know that every single one of these documents already exist so it should be getting the errors for them. At the same time the ES node is under heavy load. Could it not be returning exceptions while its under heavy load? Or is it not even attempting the create under the heavy load?
Ideally I wouldn't have to reduce the scroll size, but if thats the solution then I could.
Any help is appreciated.