Discuss the Elastic Stack

How can i process the duplication id or custome document_id on filebeat?

Elastic Stack Beats

beanslee July 5, 2017, 6:34am 1

in logstash ,i can set document_id=xxx
but in file beat， i can not find the way....

thanks!!!!

andrewkroh (Andrew Kroh) July 5, 2017, 2:34pm 2

You cannot set a document ID in Filebeat. You would need to use either Logstash or setup an Ingest Node pipeline to do it.

Topic		Replies	Views	Activity
Setting the Document ID Beats filebeat	7	1762	November 30, 2018
Deduplicate data Beats filebeat	4	569	July 15, 2022
Error calling pipeline when setting the document ID in beats Beats filebeat	7	573	June 7, 2021
Duplication in Filebeat to Elasticsearch data pushing Beats filebeat	5	761	December 28, 2017
Filebeat deduplication fail to update index Beats filebeat	8	1139	June 18, 2020

© 2020. All Rights Reserved - Elasticsearch

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.