Wrong documents' count after inserting

Elastic Stack Elasticsearch
1

Hello, please help, I inserted data with spark several times but the count was all the time bigger than expected, how it can be?
The version of ES is 8.5.0.
The query that I used to check: GET index/_count.

2

How much bigger it is?
Are you setting the id of each document or let Elasticsearch create one?
Did you run the job multiple time or are you cleaning the index before starting a new job?

3
  1. ES shows me 117 744 651 but in Spark, I have 115 925 542;
  2. ES creates id;
  3. For sure, each time before a new job I made DELETE index.
4

IMO something like a "retry" or anything like that is happening. As Elasticsearch generates the id, it creates duplicates.

You should if possible, set the id from your documents which would avoid having duplicates.

5

Thank you, that was the case.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.