# Filebeat 7.3.0 with kubernetes autodiscover and nginx module creating duplicate log entries

**URL:** <https://discuss.elastic.co/t/filebeat-7-3-0-with-kubernetes-autodiscover-and-nginx-module-creating-duplicate-log-entries/196623>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 24, 2019, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-7-3-0-with-kubernetes-autodiscover-and-nginx-module-creating-duplicate-log-entries/196623 "2019-08-24T14:35:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrisferry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisferry/32/51835_2.png) [@chrisferry](https://discuss.elastic.co/u/chrisferry)\
**Post date:** [August 24, 2019, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-7-3-0-with-kubernetes-autodiscover-and-nginx-module-creating-duplicate-log-entries/196623/1 "2019-08-24T14:35:44Z")

</div>

Running filebeat 7.3.0 and receiving duplicate log entries with two differences.  
Specifically  
fileset.name: access|error  
event.dataset nginx.access|nginx.error

The log is identical otherwise.  
Watching the docker logs I see only one entry which leads me to believe filebeat is duplicating the log as it sends.

filebeat.yml:

```
    filebeat.autodiscover:
      providers:
      - hints.default_config:
          fields:
            env: prod
          paths:
          - /var/log/containers/*${data.container.id}.log
          type: container
        hints.enabled: true
        include_annotations:
        - elk/logging
        type: kubernetes
    filebeat.config:
      inputs:
        enabled: true
        path: /etc/filebeat/*.yml
        reload.enabled: true
        reload.period: 10s
      modules:
        path: /etc/filebeat/modules.d/*.yml
        reload.enabled: false
    http.enabled: true
    http.port: 5066
    output:
      elasticsearch:
        enabled: false
        hosts:
        - http://elasticsearch-master:9200
      file:
        enabled: false
      logstash:
        enable: true
        hosts: logstash.mydomain.com:5044
        ssl:
          certificate: /tmp/filebeat/client.pem
          certificate_authorities:
          - /tmp/filebeat/root-ca.pem
          key: /tmp/filebeat/client.key
          supported_protocols:
          - TLSv1.2
          verification_mode: none
    output.file:
      filename: filebeat
      number_of_files: 5
      path: /usr/share/filebeat/data
      rotate_every_kb: 10000
    processors:
    - add_cloud_metadata: null
    - add_host_metadata:
        netinfo.enabled: true
    - add_kubernetes_metadata:
        annotations.dedot: true
        in_cluster: true
        labels.dedot: true
    - drop_event:
        when:
          equals:
            kubernetes.container.name: filebeat

```

Container annotations:

```
Annotations: co.elastic.logs.nginx/fileset.stderr: error
                co.elastic.logs.nginx/fileset.stdout: access
                co.elastic.logs.nginx/module: nginx

```

Ingestion takes place with logstash.  
The input\_beats config:

```
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => '/opt/logstash/server/config/node.pem'
    ssl_certificate_authorities => ['/opt/logstash/server/config/root-ca.pem']
    ssl_key => '/opt/logstash/server/config/node.key'
    tls_min_version => 1.2
  }
}

filter {
  if [message] =~ "^\{.*\}[\s\S]*$" {
    json {
      source => "message"
    }
  }
  if [service][type] == "nginx" {
    grok {
      match => { "message" => "%{NGINX_ACCESS}" }
    }
    geoip {
      source => "clientip"
    }
  }
  if [type] != "heartbeat" {
    mutate {
      remove_field => ["[host]" ]
    }
    mutate {
      add_field => {
        "host" => "%{[agent][hostname]}"
      }
    }
  }
}

```

Logstash: 7.3.0  
Do I have a configuration issue that is causing the nginx module to duplicate the log entries? Could it be an autodiscover bug? Should I just not use the nginx module? Seems I can tag the logs and parse them on the logstash side without even engaging the nginx module. Any advice would be appreciated. Please let me know if I can provide any additional information to aid in diagnosing this issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 21, 2019, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-7-3-0-with-kubernetes-autodiscover-and-nginx-module-creating-duplicate-log-entries/196623/2 "2019-09-21T14:35:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.