Bulk create silently failing


#1

Elasticsearch 2.4.4
Hi,

I have a script where I'm creating trying to create a large number of documents in an index that already exist, testing something before it goes in production. I have a scroll size of 50k, number of documents being written is 3 million though this will need to scale to hundreds of millions.

Most of the 50k scroll are returning 409 document already exists exception for every document in the scroll. But sometimes its returning less than it should, sometimes less than half. And I know that every single one of these documents already exist so it should be getting the errors for them. At the same time the ES node is under heavy load. Could it not be returning exceptions while its under heavy load? Or is it not even attempting the create under the heavy load?

Ideally I wouldn't have to reduce the scroll size, but if thats the solution then I could.

Any help is appreciated.


#2

Just an update, I've reduced the scroll size from 50k to 10k however I'm still seeing the issue where its not correctly reporting all the document already exists exceptions. So I'm not convinced that scroll size is the answer.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.