# Why I have log duplication when filebeat read rotating log files and stop filebeat for specific time manually and default close inactive reached **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 19, 2022, 7:16am UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663 "2022-09-19T07:16:04Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\ **Post date:** [September 19, 2022, 7:16am UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/1 "2022-09-19T07:16:04Z")
I set filebeat to read logs from rotating logs (rotated when 5 mg is reached) and below is my config : ``` - type: log fields: source: 'filebeat2' logID: logbackup fields_under_root: true enabled: true paths: - /home/logbackup/a.log -/home/logbackup/backup/a.log-* output.logstash: hosts: ["ip:5044"] worker: 4 bulk_max_size: 4096 queue: mem: events: 16384 ``` and logstash.yml : ``` pipeline.workers: 4 pipeline.batch.size: 4096 ``` and close-inactive is default(5min).we have 100 transaction per second .I stop filebeat manually for specific time(for crash test) and when start it manually (with 2 million docs stored in second directory path )and some logs been duplicated. what is the solution ,is it possible solution to increase close-inactive time ? ---
**Author:** ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)\ **Post date:** [September 21, 2022, 3:17pm UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/2 "2022-09-21T15:17:20Z")
Hello @alex_petrov It might be related to the log rotation strategies, more details on it you can find in [article](https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html), do you use log rotation strategies that copy and truncate the input log file? ---
**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\ **Post date:** [September 24, 2022, 7:06am UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/3 "2022-09-24T07:06:57Z")
I dont use this strategy ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 24, 2022, 12:56pm UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/4 "2022-09-24T12:56:22Z")
Log duplication may still happen independent of the log rotate strategy you use or any other setting you change. One thing that helps avoiding log duplication in most of the cases is to use a custom `_id` value instead of letting Elasticsearch set the `_id` value. But depending on how you are indexing your data (if you are using time based indices, if you are using data streams, if you are using rollover etc) you still may get some duplicate in some cases. ---
**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\ **Post date:** [September 26, 2022, 6:02am UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/5 "2022-09-26T06:02:50Z")
thanx.do you have any tips to set custom \_id value ? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 4, 2022, 12:43pm UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/6 "2022-10-04T12:43:49Z")
It depends entirely on your document, if you have a field that have a unique id, you can use this field to generate the custom id. Check this [part of the documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-deduplication.html#_how_can_i_avoid_duplicates) on how to deduplicate data in filebeat. ---
**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\ **Post date:** [October 8, 2022, 4:58am UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/7 "2022-10-08T04:58:17Z")
Thanx leandrojmp , I used fingerprint processor and message field in filebeat, and All message have unique value , is that right ? ``` processors: - fingerprint: fields: ["message"] method: sha256 target_field: "@metadata._id" ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 5, 2022, 6:59am UTC](https://discuss.elastic.co/t/why-i-have-log-duplication-when-filebeat-read-rotating-log-files-and-stop-filebeat-for-specific-time-manually-and-default-close-inactive-reached/314663/8 "2022-11-05T06:59:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.