# \[Filebeat\] duplicate output when using queue.disk and restarting **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 29, 2022, 8:54am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959 "2022-07-29T08:54:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Yos](https://avatars.discourse-cdn.com/v4/letter/y/ed8c4c/32.png) [@Yos](https://discuss.elastic.co/u/Yos)\ **Post date:** [July 29, 2022, 8:54am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959/1 "2022-07-29T08:54:08Z")
I am facing a problem with duplicate output when using queue.disk with Filebeat and restarting. The contents of filebeat.yml and the operating procedure are as follows filebeat.yml ```auto filebeat.inputs: - type: filestream id: test-id enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /tmp/test.log output.file: path: "/tmp/" filename: result permissions: 0640 processors: - drop_fields: fields: ["agent.id", "agent.type", "agent.version", "agent.hostname", "agent.ephemeral_id", "log.offset"] logging.level: debug logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 30 permissions: 0640 monitoring.enabled: false monitoring.cluster_uuid: "XXXXXXXXXXXXXXXXXXXXX" path.data: /data/filebeat/ http.enabled: true queue.disk: max_size: 1GB filebeat.shutdown_timeout: 30s ``` 1.Enter the first log in the target file. /tmp/test.log `2022-07-29 09:00:00.000 0000001` /tmp/result `{"@timestamp": "2022-07-29T07:41:53.980Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "input":{"type": "filestream"}, "agent":{"name": "XXXXX01"}, "ecs":{"version": "1. 12.0"}, "host":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:00:00.000 0000001"}` /data/filebeat/diskqueue ```auto Total 8 -rw------- 1 root root 216 Jul 29 16:41 0.seg -rw------- 1 root root 28 Jul 29 16:41 state.dat ``` 2.Enter the second log in the target file /tmp/test.log ```auto 2022-07-29 09:00:00.000 0000001 2022-07-29 09:01:00.000 0000002 ``` /tmp/result ```auto {"@timestamp": "2022-07-29T07:41:53.980Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "input":{"type": "filestream"}, "agent":{"name": "XXXXX01"}, "ecs":{"version": "1. 12.0"}, "host":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:00:00.000 0000001"} {"@timestamp": "2022-07-29T07:43:37.989Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "ecs":{"version": "1.12. 0"}, "host":{"name": "XXXXX01"}, "agent":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:01:00.000 0000002", "input":{"type": "filestream"}} ``` /data/file-beat/disk-queue ```auto Total 8 -rw------- 1 root root 424 Jul 29 16:43 0.seg -rw------- 1 root root 28 Jul 29 16:43 state.dat ``` 3.Restart filebeat `# sudo systemctl restart filebeat.service` 4.Enter the third log in the target file /tmp/test.log ```auto 2022-07-29 09:00:00.000 0000001 2022-07-29 09:01:00.000 0000002 2022-07-29 09:02:00.000 0000003 ``` /tmp/result `{"@timestamp": "2022-07-29T07:46:13.104Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "message": "2022-07-29 09:02:00. 000 0000003", "input":{"type": "filestream"}, "ecs":{"version": "1.12.0"}, "host":{"name": "XXXXX01"}, "agent":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}` /tmp/result.1 ```auto {"@timestamp": "2022-07-29T07:41:53.980Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "input":{"type": "filestream"}, "agent":{"name": "XXXXX01"}, "ecs":{"version": "1. 12.0"}, "host":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:00:00.000 0000001"} {"@timestamp": "2022-07-29T07:43:37.989Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "ecs":{"version": "1.12. 0"}, "host":{"name": "XXXXX01"}, "agent":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:01:00.000 0000002", "input":{"type": "filestream"}} ``` /data/file-beat/disk-queue ```auto Total 12 -rw------- 1 root root 424 Jul 29 16:43 0.seg -rw------- 1 root root 216 Jul 29 16:46 1.seg -rw------- 1 root root 28 Jul 29 16:46 state.dat ``` **Segments are divided** 5.Restart filebeat `# sudo systemctl restart filebeat.service` /tmp/result `{"@timestamp": "2022-07-29T07:46:13.104Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "log":{"file":{"path":"/tmp/test. log"}}, "message": "2022-07-29 09:02:00.000 0000003", "input":{"type": "filestream"}, "ecs":{"version": "1.12.0"}, "host":{"name": "XXXXX01"}, "agent":{"name": "XXXXX01"}}` **[DUPLICATED]Same contents as /tmp/result.1** /tmp/result.1 `{"@timestamp": "2022-07-29T07:46:13.104Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "message": "2022-07-29 09:02:00. 000 0000003", "input":{"type": "filestream"}, "ecs":{"version": "1.12.0"}, "host":{"name": "XXXXX01"}, "agent":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}` /tmp/result.2 ```auto {"@timestamp": "2022-07-29T07:41:53.980Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "input":{"type": "filestream"}, "agent":{"name": "XXXXX01"}, "ecs":{"version": "1. 12.0"}, "host":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:00:00.000 0000001"} {"@timestamp": "2022-07-29T07:43:37.989Z","@metadata":{"beat": "filebeat", "type":"_doc", "version": "7.17.5"}, "ecs":{"version": "1.12. 0"}, "host":{"name": "XXXXX01"}, "agent":{"name": "XXXXX01"}, "log":{"file":{"path":"/tmp/test.log"}}, "message": "2022-07-29 09:01:00.000 0000002", "input":{"type": "filestream"}} ``` After this, "2022-07-29 09:02:00.000 0000003" is output every time filebeat is restarted. Is this correct behavior? Also, please point out if there are any mistakes in the settings, etc. ---
**Author:** ![Yos](https://avatars.discourse-cdn.com/v4/letter/y/ed8c4c/32.png) [@Yos](https://discuss.elastic.co/u/Yos)\ **Post date:** [August 1, 2022, 1:28am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959/2 "2022-08-01T01:28:30Z")
Hmmm, I get the same result no matter how many times I try. Is this correct behavior? Or is it a bug? ---
**Author:** ![Yos](https://avatars.discourse-cdn.com/v4/letter/y/ed8c4c/32.png) [@Yos](https://discuss.elastic.co/u/Yos)\ **Post date:** [August 1, 2022, 8:49am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959/3 "2022-08-01T08:49:13Z")
> [@Yos](#): > > ```auto > queue.disk: > max_size: 1GB > > ``` If these two lines are commented out and a memory queue is used, this phenomenon does not occur. ---
**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\ **Post date:** [August 1, 2022, 9:35am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959/4 "2022-08-01T09:35:27Z")
Hi @Yos, Thanks for noticing and reporting this. Indeed it seems to be a bug with the disk queue. Thanks a lot for the detailed setp-by-step to reproduce the bug! I'll open a issue on our GitHub today. ---
**Author:** ![Yos](https://avatars.discourse-cdn.com/v4/letter/y/ed8c4c/32.png) [@Yos](https://discuss.elastic.co/u/Yos)\ **Post date:** [August 2, 2022, 12:12am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959/5 "2022-08-02T00:12:07Z")
Hi @TiagoQueiroz , Thanks for your response and opening a issue([32560](https://github.com/elastic/beats/issues/32560)) on our GitHub. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2022, 2:12am UTC](https://discuss.elastic.co/t/filebeat-duplicate-output-when-using-queue-disk-and-restarting/310959/6 "2022-08-30T02:12:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.