# Elasticsearch documents getting deleted

**URL:** <https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854>\
**Category:** Elasticsearch\
**Created:** [September 10, 2025, 7:53pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854 "2025-09-10T19:53:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ItsHoney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itshoney/32/144983_2.png) [@ItsHoney](https://discuss.elastic.co/u/ItsHoney)\
**Post date:** [September 10, 2025, 7:53pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854/1 "2025-09-10T19:53:10Z")

</div>

We’ve been experiencing an issue where certain documents in our `Media` index go missing. What’s unusual is that it’s often the _same documents_ that disappear each time.

Here’s what we’ve observed:

- We have a process that deletes older media when new media is indexed.

- Initially, we suspected that deletes might be happening **after** new documents were indexed, but the behavior is inconsistent.

- Because of this, we’re not sure if the issue is tied to our delete logic or something else.

We also tried enabling audit logs to investigate further, but we’re struggling with filtering. Specifically:

- We delete documents based on an `ExpandKey` field.

- We’d like the cluster to log **only delete events** where `ExpandKey` starts with a certain prefix, instead of logging _all_ delete operations (which creates a lot of noise).

We’d really appreciate guidance on:

1. Best practices for tracking down unexpected deletions in Elasticsearch/OpenSearch.

2. How to configure audit logging (or another mechanism) to capture _only the deletes that match certain field criteria_.

P.S I have realized that we are using routing when indexing documents, but not using routing when deleting them. I will try to rectify this, but can this be a cause of this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2025, 7:53pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854/2 "2025-09-10T19:53:10Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See [What is OpenSearch and the OpenSearch Dashboard? | Elastic](https://www.elastic.co/elasticsearch/opensearch) for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2025, 2:04pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854/3 "2025-09-11T14:04:16Z")

</div>

> [@ItsHoney](#):
>
> Best practices for tracking down unexpected deletions in Elasticsearch/OpenSearch.

Which product are you using? It is quite possible that the answer depends on this.

> [@ItsHoney](#):
>
> Because of this, we’re not sure if the issue is tied to our delete logic or something else.

> [@ItsHoney](#):
>
> We delete documents based on an `ExpandKey` field.

It sounds like you might be using delete by query to remocve old data. It would be useful to have more detail about exactly how your deletion process works.

> [@ItsHoney](#):
>
> How to configure audit logging (or another mechanism) to capture _only the deletes that match certain field criteria_.

This will depend a lot on the product and version used, which you have not specified.

> [@ItsHoney](#):
>
> P.S I have realized that we are using routing when indexing documents, but not using routing when deleting them. I will try to rectify this, but can this be a cause of this?

That depends on how you delete data.

---

<div class="post-metadata">

**Author:** ![ItsHoney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itshoney/32/144983_2.png) [@ItsHoney](https://discuss.elastic.co/u/ItsHoney)\
**Post date:** [September 11, 2025, 2:16pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854/4 "2025-09-11T14:16:47Z")

</div>

1. I am using Managed OpenSearch service from AWS.

2. Yes I am using delete by query.

So for our media index, whenever we have an update for our Media, we first delete the previously existing Media documents using an ExpandKey.

Here’s an example,

`/media_1_7/_delete_by_query` `{"query":{"query_string":{"query":"Metadata.ExpandKey.keyword: 13711637256"}}}`

The ExpandKey is a combination of {datasourceID}{listingID}. so `{1371}{1637256}`

For different data sources, we can have the same ListingID. But this combination should be unique in the whole system.

So before indexing the new media, we have code to delete older media like so:

```auto
        indexRecords = indexRecords.Where(doc => doc.DoIndex).ToList();
        // run a delete before indexing the documents
        var deleteResponses = await indexRecords
            .Where(x => x.Indexing.DeleteQuery != null)
            .Select(_esRepo.DeleteDocumentAsync)
            .WhenAll()
            .ConfigureAwait(false);
        foreach (var error in deleteResponses.Where(x => x.OriginalException != null))
        {
// log errors
        }

        // now index all the records
        var bulkResponse = await _esRepo.IndexDocumentsAsync(indexRecords);

```

1. I am using OpenSearch version OpenSearch\_2\_11\_R20250630

I hope it answers your questions!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2025, 2:49pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854/5 "2025-09-11T14:49:43Z")

</div>

> [@ItsHoney](#):
>
> I am using Managed OpenSearch service from AWS.

OpenSearch is a different product from Elasticsearch so I would recommend you reach out to the OpenSearch community or AWS support. Their implementation of security and audit logging is completely different from Elasticsearch and I do not know whether there are any special limitations or peculiarities related to their managed service.

> [@ItsHoney](#):
>
> The ExpandKey is a combination of {datasourceID}{listingID}. so `{1371}{1637256}`

Are the fields fixed length or is it possible `{1371}{1637256}` could exist at the same time as e.g. `{137}{11637256}`?

---

<div class="post-metadata">

**Author:** ![ItsHoney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itshoney/32/144983_2.png) [@ItsHoney](https://discuss.elastic.co/u/ItsHoney)\
**Post date:** [September 11, 2025, 6:12pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-deleted/381854/6 "2025-09-11T18:12:05Z")

</div>

I think you might’ve caught the problem! I am so shocked I didn’t consider it xD

Thanks alot! I think the problem has mostly been resolved