# Elastic deleted documents

**URL:** <https://discuss.elastic.co/t/elastic-deleted-documents/344097>\
**Category:** Elasticsearch\
**Created:** [September 29, 2023, 4:48am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097 "2023-09-29T04:48:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [September 29, 2023, 4:48am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/1 "2023-09-29T04:48:14Z")

</div>

All the documents which are ingested by logstash are deleted automatically. Please find the screen shot. I do not have any ISM policy to delete document.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/1/3162097beb9da0e74d929c82fb42bc7ab6613bd3.png)

Any help!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 29, 2023, 5:31am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/2 "2023-09-29T05:31:34Z")

</div>

How are uou indexing data into Elasticsearch? Are you by any chance setting a document id which happens to be the same for all documents (check the ID of the document you have in the index and verify that it is what you expect)?

---

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [September 29, 2023, 7:57am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/3 "2023-09-29T07:57:55Z")

</div>

@Christian_Dahlqvist Thanks for helping.  
In the output configuration I specify document\_id =\> "%{fingerprint}"  
In the filter I have

```auto
filter {
  fingerprint {
      method => "SHA1"
      key => "787878"
  }
}

```

The latest logstash did not replace the value for fingerprint ("%{fingerprint}")  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/938975cfb8af27b4f75bf5d3e288fbb14a7844b0.png)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 29, 2023, 8:03am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/4 "2023-09-29T08:03:43Z")

</div>

It looks like the fingerprint field is not populated for any event, which indicates that you have misconfigured the fingerprint filter.

---

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [September 29, 2023, 8:14am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/5 "2023-09-29T08:14:18Z")

</div>

@Christian_Dahlqvist Appreciate your help on this regard.  
I shared the configuration with respect to fingerprint. If you would let me know why fingerprint is not getting populated. It would be helpful.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 29, 2023, 8:23am UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/6 "2023-09-29T08:23:31Z")

</div>

I have not used this plugin in a long while, but it seems you may need to specify a target or check whether ECS is enabled and the fingerprint instead is written to the `"[event][hash]"` field.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 29, 2023, 12:10pm UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/7 "2023-09-29T12:10:21Z")

</div>

> [@ranjini](#):
>
> I shared the configuration with respect to fingerprint. If you would let me know why fingerprint is not getting populated. It would be helpful.

Please share the rest of your pipeline, just this is not enough.

The `fingerprint` field uses a field as a source to create a fingerprint, if you do not specify any field, it will use the `message` field of you event, if you do not have a `message` field in your event I think that it will probably not work, which seems to be the case.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2023, 12:11pm UTC](https://discuss.elastic.co/t/elastic-deleted-documents/344097/8 "2023-10-27T12:11:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.