Nest BulkAll indexing duplicates

Elastic Stack Elasticsearch
1

Hi all,

I'm indexing a bunch of documents using Nest and the BulkAll method like that:

var waitHandle = new CountdownEvent(1);

var bulkAll = _client.BulkAll(elementList, b => b
.Index(indexName)
.BackOffRetries(15)
.BackOffTime(TimeSpan.FromSeconds(55))
.RefreshOnCompleted()
.MaxDegreeOfParallelism(4)
.Size(1500));

bulkAll.Subscribe(observer: new BulkAllObserver(
onNext: (b) =>
{
_logger.Debug("Indexed group of documents");
},
onError: (e) =>
{
_logger.Error(e, e.Message);
throw e;
},
onCompleted: () =>
{
waitHandle.Signal();
}));
waitHandle.Wait();

The problem is that sometimes it fails indexing a block of documents and retries automatically, generating duplicates.

I'm supposed to have 250000 documents (for example), but once that the index is completed, it contains 251500, 253000 or even more.

How can I use the retry logic but avoiding duplicates?

Thanks!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.