Elasticsearch documents deleted automatically

Elastic Stack Elasticsearch
1

Hi Team,

In our production servers, documents are missing particular type and particular shard.

same type other shard data available.
no shard relocation, no node / cluster restart happen at the time. even no log messages printed on that time.

Any clue on this? what i need to do now, please advice.

2

i see here deleted documents increased, and number of documents reduced in a particular time. nearly 1 lack documents deleted.

we sure there isno manual delete happen, through api or from application layer.

below the graphs shows deleted count and document count ( happen at 15:34:12 )

3

ES will only delete documents if

  1. Something makes a delete request
  2. Something updates an existing document
4

HI warkolm,

thanks for reply,

We verified our server logs , history and application logs there is NO MANUAL deletion,
is there any option to check how the documents are deleted ( i.e through api or from app layer )
let me know which logs help for me.

5

The only way would be if you have a proxy that catches all the requests, or something like Shield that does audit logging.

6

ok, we are using 1.3.7 version

is there any mapping conflict can cause this issue,
i see mapping conflicts on that index.

7

That's a really old version, you should upgrade immediately.

8

is there any mapping conflict can cause this issue,
i see mapping conflicts on that index.

9

_ttl will also delete documents but you have to turn it on and you probably
didn't. I expect there are some log levels you can crank up to get all the
deleted documents to be logged but I'd have to look around to figure out
what they are and I'm not in a place I can do that. Thatd only tell you
about the next delete, it'd probably tell you about every index too, and I
doubt it'd tell you why the delete happened, just that it happened.

Beyond that I don't know what to say.

10

we don't have _ttl ,

i don't know why that particular type in a particular shard data missing,
it occurs on another cluster also.

11

Then something is deleting them.

12

Hi @warkolm @nik9000

Many more thanks, we identified the issue in our app level.
But we need to setup the query logging ( search, delete, index ).
we are working on that. Again Thanks lot.