Hi Team, I am using a solution that reads JSON logs by filebeat and push the data to ES without logstasg. While I am checking I can see duplication in documentation. Could you please let us know how can we configure a uniq id (similar to the document id in logstash "document_id => "%{eventid}-%{t…

No ETA. Some related work is done in the PRs #5811 and #5844 , but there is still some more work required for supporting setting document IDs in beats.

Discuss the Elastic Stack

Duplication in Filebeat to Elasticsearch data pushing

Elastic Stack Beats

john.akash (Akash John) December 7, 2017, 7:25pm 3

Could you please provide us on ETA on that?

Topic		Replies	Views
Filebeat and updating documents Beats filebeat	2	1166	March 7, 2019
Duplicated events Beats filebeat	3	347	September 22, 2023
Filebeat deduplication fail to update index Beats filebeat	8	1130	June 18, 2020
Deduplicate data Beats filebeat	4	567	July 15, 2022
How to use document id to avoid duplication of logs? Logstash	8	2068	June 26, 2020

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Duplication in Filebeat to Elasticsearch data pushing

Related topics