Filebeat autodiscovery logging twice

[strowi (Roman)]

Hi,

i'm currently playing with hint-based autodiscovery in kubernetes. So far my tests with an nginx-deployment seem to work fine, except that the logline appears twice in elasticsearch.
One time as "message", the other time as nginx-parsed output.

Is there a way to only log the parsed log with hint-based discovery?

regards,
strowi

[kvch (Noémi Ványi)]

Could you please share your configuration formatted using </>?

[strowi (Roman)]

Here is the one from filebeat:

logging.level: 'debug'
tags: ['k8-rancher2-tier']
name: ${FILEBEAT.NAME}

output.elasticsearch:
  index: "logs-%{+yyyy.MM.dd}"

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  inputs:
    path: ${path.config}/prospectors.d/*.yml
    reload.enabled: false
    ignore_older: 24h

setup:
  template:
    name: "logs"
    pattern: "logs-*"

  kibana:
    host: 

  dashboards:
    enabled: true
    index: "logs-*"

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /var/log/messages
  - /var/log/syslog

- type: docker
  containers.ids:
  - "*"
  processors:
    - add_kubernetes_metadata:
        in_cluster: "true"
    - drop_event:
        when:
          regexp:
            kubernetes.container.name: ".*(heapster|kube-state-metrics).*"
    - drop_event:
        when:
          not:
            regexp:
              kubernetes.pod.name: ".+"


filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

and an example nginx-deployment:

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    team: roman
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        co.elastic.logs/module: nginx
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/fileset.stderr: error
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80

[kvch (Noémi Ványi)]

Where does your nginx put your logs? Have you enabled the nginx module?
It is possible that your logs are duplicated, because you have configured both the nginx module and the log input of Filebeat to read the logs. If yes, please make sure that you only configure a file once, otherwise it can lead to problems in state handling.
You can validate my theory by checking if the duplicated events have fileset.module and fileset.name keys with values.