# Filebeat duplicate logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 12, 2022, 12:53pm UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032 "2022-12-12T12:53:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 12, 2022, 12:53pm UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/1 "2022-12-12T12:53:53Z")
Hello everyone, for prevent the duplication data that can be received from Filebeat I used this Logstash filtration fingerprint { source =\> "message" target =\> "[@metadata][fingerprint]" method =\> "SHA1" key =\> "key" base64encode =\> true } but if appears duplicate inside log file this filtration also prevent - I need to prevent duplicate data that can happen from some cases like filebeat restart but if log file there is in it data duplication i want to permit this thank you ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2022, 1:07pm UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/2 "2022-12-12T13:07:28Z")
Are you using the fingerprint as document ID in your Elasticsearch output? Are you indexing into time-based indices based on rollover? ---
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 13, 2022, 7:16am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/3 "2022-12-13T07:16:16Z")
yes, I don't have problems in the configurations Logstash pipeline but the problem in how to prevent the duplicate data if case happens like filebeat restart without prevent the duplication if happens from the logs ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 13, 2022, 7:22am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/4 "2022-12-13T07:22:22Z")
I do not understand what you mean. Can you please elaborate? ---
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 13, 2022, 7:44am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/5 "2022-12-13T07:44:19Z")
some cases can cause the data duplication like filebeat restarting for control in this case I used fingerprint filtration but this approach prevent everything of duplication even if the log file there is duplicate of data. I want to make if the duplication from filebeat or everything it is prevent but if it from log file i want to permit ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 13, 2022, 7:59am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/6 "2022-12-13T07:59:57Z")
> [@m3bgwad](#): > > some cases can cause the data duplication like filebeat restarting for control in this case I used fingerprint filtration but this approach prevent everything of duplication even if the log file there is duplicate of data. Filebeat should be able to handle restarts without duplicating a lot of data. What type of storage are you reading from? How is your Filebeat configured? > [@m3bgwad](#): > > I want to make if the duplication from filebeat or everything it is prevent but if it from log file i want to permit In your example you are calculating a fingerprint based on the contents of the log line. Identical log lines in log files will therefore result in the same fingerprint and cause updates in Elasticsearch. You could add the filename to the string you use to determine the fingerprint and this would allow the same log line from different files to be inserted without resulting in updates. ---
**Author:** ![m3bgwad](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Post date:** [December 13, 2022, 8:29am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/7 "2022-12-13T08:29:06Z")
if in the same log file there is duplications, what is the solution? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 13, 2022, 8:46am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/8 "2022-12-13T08:46:57Z")
Filebeat can add/adds [the offset for each log line](https://www.elastic.co/guide/en/beats/filebeat/8.5/exported-fields-log.html) so you could include this when calculating the fingerprint. I do not believe the Logstash file input plugin is able to do this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 10, 2023, 8:47am UTC](https://discuss.elastic.co/t/filebeat-duplicate-logs/321032/9 "2023-01-10T08:47:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.