How to handle duplicate records in datastreams using fingerprint

[Tortoise]

Hello @venkatkumar229

Data streams are designed for append-only time-series data so they are not suitable when you need global de-duplication based on _id.
_id uniqueness is enforced only within a single backing index so after rollover the same _id can be indexed again in a new backing index (the same behavior seen in your case)

The same thing happens with regular indices using rollover and a write alias because Elasticsearch does not check older indices for existing _ids.

If you require exactly one document per business key (for example seqId), the recommended approach is to use a Transform or will have to avoid rollover of index (which might not be a feasible solution).

Similar post :

Fingerprint processor allowing duplicates

Yes, it is really an issue, as it's skewing metrics, and duplicates will continue to happen each time the index rolls over. It's a timing issue with the source data in SQL. The write interval is in some cases longer than the query interval. The SQL query intentionally returns the same records multiple times (with some additional each time), to ensure there's eventually a superset of the data. It's not ideal, but they're trying to take the weight of the query off the SQL database, so I've been r…

Thanks!!