Unexpected duplicate documents created by Elastic Search

Elastic Search
1

I use the BulkAll C# ElasticClient API to create a new index of ~500,000 documents everyday. No other documents are added to this index after this initial creation. In my code, I also immediately check the Count C# ElasticClient API after the BulkAll call to confirm the correct number of documents were indexed. The problem I'm having is that, while this initial Count returns the expected number of documents as being in the index, some time later the number of documents in the index becomes around 2,000-3,000 MORE than that originally expected and confirmed number. And random searching identifies that some of the documents in this index are stored twice (identical duplicates). How could this happen? I'm more looking for either some better way I could be using the BulkAll C# API to prevent this from happening or some way to determine why or what is causing elasticsearch to create these duplicates after an apparent confirmation of no duplicates.

elasticsearch version 8.15.3
NEST client version 7.17.5

indexing code:

                var ElasticConn = new ElasticClient(s);
                ElasticConn.BulkAll(docArray,
                     b => b
                        .MaxDegreeOfParallelism(4)
                        .BulkResponseCallback((r) => 
                        {
                            bulkResponses.Add($"{(r.OriginalException != null ? r.OriginalException.ToString() : string.Empty)}|{(r.ServerError != null ? r.ServerError.Error.ToString() : string.Empty)}|DEBUG: {r.DebugInformation}");
                        })
                        .DroppedDocumentCallback((r, o) =>
                        {
                            if (r != null && r.Error != null)
                            {
                                droppedDocumentReasons.Add(r.Error.Reason);
                            }
                            else
                            {
                                droppedDocumentReasons.Add(r.Result);
                            }
                        })
                        .BackOffTime("30s")
                        .BackOffRetries(5)
                        .Size(1000)
                        .RefreshOnCompleted()
                        .Index("IndexName")
                        );

None of the DroppedDocumentCallback or BulkResponseCallback functions seem to be getting called

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.