Loosing data in elasticsearch

This makes sense, this would be from the kafka elasticsearch connector as I don't have any delete function enabled.

Here is the index setting:


Can't find anything on ttl

That is good, because it was deprecated for a reason. I notice that you have a lot of nested fields. As each of the nested documents is stored as a separate document in Elasticsearch and all documents are updated when the main document is, it can mean that updating a single document will lead to multiple documents being updated and shown as deleted in the stats.

For reference, related discussion : https://stackoverflow.com/questions/45018344/using-elasticsearch-generated-ids-in-kafka-elasticsearch-connector/45182181#45182181

Thanks @Christian_Dahlqvist and @warkolm, It seems the problem has been resolved by changing to a different elasticsearch connector.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.