# Filebeat duplicate log
**URL:** https://discuss.elastic.co/t/filebeat-duplicate-log/224209
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [March 19, 2020, 2:07am UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209 "2020-03-19T02:07:25Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@ZPerling](https://discuss.elastic.co/u/ZPerling)
#### Post date: [March 19, 2020, 2:07am UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209/1 "2020-03-19T02:07:25Z")
So, my filebeat running in the kubernetes cluster as daemonsets. and filebeat config:
```auto
logging.level: info
path.home: "/usr/share/filebeat"
path.config: "/usr/share/filebeat"
path.data: "/usr/share/filebeat/data"
path.logs: "/usr/share/filebeat/logs"
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
has_fields: ["kubernetes.labels.kafkaTopic"]
config:
- type: log
enabled: true
paths:
- /data/logs/${data.kubernetes.labels.service}-${data.kubernetes.labels.cluster}_${data.kubernetes.namespace}/${data.kubernetes.pod.name}/*/*.log
- type: log
enabled: true
symlinks: true
json.keys_under_root: false
paths:
- /var/log/pods/${data.kubernetes.namespace}_${data.kubernetes.pod.name}_${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/*.log
processors:
- rename:
fields:
- from: "json.log"
to: "message"
- from: "json.stream"
to: "stream"
- from: "json.time"
to: "datetime"
ignore_missing: false
fail_on_error: true
- drop_fields:
fields: ["json"]
processors:
- if:
regexp:
message: "{.*}"
then:
- rename:
fields:
- from: "message"
to: "message_json_str"
ignore_missing: false
fail_on_error: true
- decode_json_fields:
fields: ["message_json_str"]
process_array: true
max_depth: 5
target: ""
overwrite_keys: false
add_error_key: true
- drop_fields:
fields: ["message_json_str"]
- rename:
fields:
- from: "log.file.path"
to: "log_path"
- from: "kubernetes.replicaset.name"
to: "kubernetes.replicaset_name"
- from: "kubernetes.pod.name"
to: "kubernetes.pod_name"
- from: "kubernetes.node.name"
to: "kubernetes.node_name"
- from: "host.name"
to: "fagent"
ignore_missing: false
fail_on_error: true
- drop_fields:
fields:
- "kubernetes.container"
- "kubernetes.replicaset"
- "kubernetes.pod"
- "kubernetes.node"
- "kubernetes.labels.pod-template-hash"
- "agent"
- "ecs"
- "log"
- "input"
- "host"
output.kafka:
enabled: true
hosts: '${KAFKA_HOSTS}'
topic: "%{[kubernetes.labels.kafkaTopic]}"
partition.round_robin:
reachable_only: true
required_acks: 1
compression: gzip
max_message_bytes: 1000000
channel_buffer_size: 1024
keep_alive: 60
client_id: ${HOSTNAME:beats}
worker: 3
```
Now I test write "test" into log file use cmd "echo test \>\> test.log", kafka topic received eight count same log, I do not know what happened, anyone can help you? thanks.
---
### Author:  [@ZPerling](https://discuss.elastic.co/u/ZPerling)
#### Post date: [March 19, 2020, 2:09am UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209/2 "2020-03-19T02:09:07Z")
I found filebeat create eight harvester for one log file:
```auto
2020-03-16T06:05:38.211Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.211Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.392Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.392Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.392Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.572Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.677Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
2020-03-16T06:05:38.815Z INFO log/harvester.go:251 Harvester started for file: /data/logs/office-api-stag_op-stag/office-api-stag-7cb9cbfb44-6ll55/office-api/test.log
```
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [March 19, 2020, 11:43am UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209/3 "2020-03-19T11:43:07Z")
Could you please share the debug logs of Filebeat (`./filebeat -e -d "*"`)? It is possible something is misconfigured. Filebeat should not start more than one harvester for a file, because it leads to issues with state handling.
---
### Author:  [@ZPerling](https://discuss.elastic.co/u/ZPerling)
#### Post date: [March 19, 2020, 2:31pm UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209/4 "2020-03-19T14:31:15Z")
Thanks for your reply. The debug log is too big which can't be pasted here directly. I've uploaded it to github. Please refer to the following link:
[https://raw.githubusercontent.com/ZPerling/filebeat\_debug\_log/master/filebeat\_debug.log](https://raw.githubusercontent.com/ZPerling/filebeat_debug_log/master/filebeat_debug.log)
---
### Author:  [@ZPerling](https://discuss.elastic.co/u/ZPerling)
#### Post date: [March 25, 2020, 8:41am UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209/5 "2020-03-25T08:41:05Z")
@kvch is there any update? 😟
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 22, 2020, 8:41am UTC](https://discuss.elastic.co/t/filebeat-duplicate-log/224209/6 "2020-04-22T08:41:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.