Elastic deleted documents

Elastic Stack Elasticsearch
1

All the documents which are ingested by logstash are deleted automatically. Please find the screen shot. I do not have any ISM policy to delete document.

image
image1403×511 31.4 KB

Any help!

2

How are uou indexing data into Elasticsearch? Are you by any chance setting a document id which happens to be the same for all documents (check the ID of the document you have in the index and verify that it is what you expect)?

3

@Christian_Dahlqvist Thanks for helping.
In the output configuration I specify document_id => "%{fingerprint}"
In the filter I have

filter {
  fingerprint {
      method => "SHA1"
      key => "787878"
  }
}

The latest logstash did not replace the value for fingerprint ("%{fingerprint}")
image

4

It looks like the fingerprint field is not populated for any event, which indicates that you have misconfigured the fingerprint filter.

5

@Christian_Dahlqvist Appreciate your help on this regard.
I shared the configuration with respect to fingerprint. If you would let me know why fingerprint is not getting populated. It would be helpful.

6

I have not used this plugin in a long while, but it seems you may need to specify a target or check whether ECS is enabled and the fingerprint instead is written to the "[event][hash]" field.

7

Please share the rest of your pipeline, just this is not enough.

The fingerprint field uses a field as a source to create a fingerprint, if you do not specify any field, it will use the message field of you event, if you do not have a message field in your event I think that it will probably not work, which seems to be the case.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.