# Filebeat+pipeline+es log duplication Help!

**URL:** <https://discuss.elastic.co/t/filebeat-pipeline-es-log-duplication-help/334406>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 26, 2023, 8:14am UTC](https://discuss.elastic.co/t/filebeat-pipeline-es-log-duplication-help/334406 "2023-05-26T08:14:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Teresajw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/teresajw/32/119937_2.png) [@Teresajw](https://discuss.elastic.co/u/Teresajw)\
**Post date:** [May 26, 2023, 8:14am UTC](https://discuss.elastic.co/t/filebeat-pipeline-es-log-duplication-help/334406/1 "2023-05-26T08:14:43Z")

</div>

When I use the following configuration of filebeat+pipeline to collect nginx logs, multiple duplicate logs will be generated in Elasticsearch, but I cannot find the reason, despite testing many times. Could you please help me see what caused the problem  
**my-filebeat.yml**

```auto
filebeat.inputs:
- type: filestream
  paths:
  - /root/rsync_logs/nginx_logs/gate.access.log
  tags: ["gate","nginx"]
processors:
- drop_fields:
    fields:
    - agent.ephemeral_id
    - agent.hostname
    - agent.id
    - agent.type
    - agent.version
    - ecs.version
    - input.type
    - log.offset
    - version

output.elasticsearch:
  hosts: ["192.168.112.44:30200"]
  protocol: "https"
  username: "elastic"
  password: "XXXXXXXXX"
  ssl.verification_mode: none
  timezone: "Asia/Shanghai"
  enabled: true
  pipelines:
  - pipeline: "biot_nginx_log_pipeline"
    when.contains:
      tags: "nginx"
  indices:
  - index: "biot-log-nginx-gate-%{+yyyy.MM.dd}"
    when.contains:
      tags: "gate"
setup.ilm.enabled: false
setup.template.name: "biot-log"
setup.template.pattern: "biot-log*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0

```

**I can find that multiple duplicate logs are identical from the es, but my original log is only one, and the logs are still increasing repeatedly**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/22cbf70a112aae6b681b38ffbfa9ded7ed765005.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af09dadcd75c962dad47d4e60d11caad722e4742.png)

**This is my pipeline**

```auto
[
  {
    "grok": {
      "field": "message",
      "patterns": [
        "%{WORD:client_address}:%{IP:client_address}###%{WORD:client_user}:%{DATA:client_user}###%{WORD:real_ip}:%{DATA:real_ip}###%{WORD:visit_time}:%{TIMESTAMP_ISO8601:visit_time}###%{WORD:request_uri}:%{GREEDYDATA:request_uri}###%{WORD:request_host}:%{DATA:request_host}###%{WORD:http_status}:%{NUMBER:http_status}###%{WORD:upstream_status}:%{DATA:upstream_status}###%{WORD:traffic}:%{NUMBER:traffic}###%{WORD:original_address}:%{DATA:original_address}###%{WORD:http_user_agent}:%{GREEDYDATA:http_user_agent}###%{WORD:request_length}:%{NUMBER:request_length}###%{WORD:load_balancer}:%{DATA:load_balancer}###%{WORD:processing_time}:%{NUMBER:processing_time}###%{WORD:upstream_response_time}:%{DATA:upstream_response_time}###%{WORD:http_acl_t}:%{DATA:http_acl_t}###"
      ],
      "ignore_failure": true
    }
  },
  {
    "date": {
      "field": "visit_time",
      "target_field": "@timestamp",
      "formats": [
        "ISO8601"
      ],
      "timezone": "Asia/Shanghai"
    }
  },
  {
    "geoip": {
      "field": "client_address",
      "target_field": "geo",
      "ignore_failure": true
    }
  }
]

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 26, 2023, 12:23pm UTC](https://discuss.elastic.co/t/filebeat-pipeline-es-log-duplication-help/334406/2 "2023-05-26T12:23:27Z")

</div>

> [@Teresajw](#):
>
> `/root/rsync_logs/nginx_logs/`

How are you putting logs in this folder?

Its name, _rsync\_logs_, suggests that you are using rsync to copy files from this folder.

If I'm not wrong, when using `rsync` it will make a backup of the current file on the destination, transfer it from the source, and then remove the backup file.

The main issue is that this will change the inode of the file and filebeat uses the inode to track the files, if the name is the same, but the inode is different, filebeat will consider this a new file and it may lead to duplication.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 23, 2023, 2:23pm UTC](https://discuss.elastic.co/t/filebeat-pipeline-es-log-duplication-help/334406/3 "2023-06-23T14:23:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.