Filebeat 7.3.0 with kubernetes autodiscover and nginx module creating duplicate log entries

Elastic Stack Beats
1

Running filebeat 7.3.0 and receiving duplicate log entries with two differences.
Specifically
fileset.name: access|error
event.dataset nginx.access|nginx.error

The log is identical otherwise.
Watching the docker logs I see only one entry which leads me to believe filebeat is duplicating the log as it sends.

filebeat.yml:

    filebeat.autodiscover:
      providers:
      - hints.default_config:
          fields:
            env: prod
          paths:
          - /var/log/containers/*${data.container.id}.log
          type: container
        hints.enabled: true
        include_annotations:
        - elk/logging
        type: kubernetes
    filebeat.config:
      inputs:
        enabled: true
        path: /etc/filebeat/*.yml
        reload.enabled: true
        reload.period: 10s
      modules:
        path: /etc/filebeat/modules.d/*.yml
        reload.enabled: false
    http.enabled: true
    http.port: 5066
    output:
      elasticsearch:
        enabled: false
        hosts:
        - http://elasticsearch-master:9200
      file:
        enabled: false
      logstash:
        enable: true
        hosts: logstash.mydomain.com:5044
        ssl:
          certificate: /tmp/filebeat/client.pem
          certificate_authorities:
          - /tmp/filebeat/root-ca.pem
          key: /tmp/filebeat/client.key
          supported_protocols:
          - TLSv1.2
          verification_mode: none
    output.file:
      filename: filebeat
      number_of_files: 5
      path: /usr/share/filebeat/data
      rotate_every_kb: 10000
    processors:
    - add_cloud_metadata: null
    - add_host_metadata:
        netinfo.enabled: true
    - add_kubernetes_metadata:
        annotations.dedot: true
        in_cluster: true
        labels.dedot: true
    - drop_event:
        when:
          equals:
            kubernetes.container.name: filebeat

Container annotations:

Annotations:    co.elastic.logs.nginx/fileset.stderr: error
                co.elastic.logs.nginx/fileset.stdout: access
                co.elastic.logs.nginx/module: nginx

Ingestion takes place with logstash.
The input_beats config:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => '/opt/logstash/server/config/node.pem'
    ssl_certificate_authorities => ['/opt/logstash/server/config/root-ca.pem']
    ssl_key => '/opt/logstash/server/config/node.key'
    tls_min_version => 1.2
  }
}

filter {
  if [message] =~ "^\{.*\}[\s\S]*$" {
    json {
      source => "message"
    }
  }
  if [service][type] == "nginx" {
    grok {
      match => { "message" => "%{NGINX_ACCESS}" }
    }
    geoip {
      source => "clientip"
    }
  }
  if [type] != "heartbeat" {
    mutate {
      remove_field => [ "[host]" ]
    }
    mutate {
      add_field => {
        "host" => "%{[agent][hostname]}"
      }
    }
  }
}

Logstash: 7.3.0
Do I have a configuration issue that is causing the nginx module to duplicate the log entries? Could it be an autodiscover bug? Should I just not use the nginx module? Seems I can tag the logs and parse them on the logstash side without even engaging the nginx module. Any advice would be appreciated. Please let me know if I can provide any additional information to aid in diagnosing this issue.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.