Filebeat 7.2.0 autodiscover configuration issues

[dnmahendra (Mahendra)]

I am trying to setup filebeat for the kubernetes cluster. I am trying to use autodiscover for my containers. I have istio envoy proxy running on each pod. I am trying to ship those logs to my elastic stack. But, I am getting following errors -
2019-06-30T07:05:21.816Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: Fileset envoyproxy/access is configured but doesn't exist
2019-06-30T07:05:21.816Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: each input must have at least one path defined

Here is my yaml config I using to setup filebeat,

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    setup.dashboards.enabled: true
    setup.template.enabled: true
    setup.kibana.host: "192.168.82.162:5601"

    setup.template.settings:
      index.number_of_shards: 1
    setup.template.overwrite: true
    filebeat.overwrite_pipelines: true


    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          templates:
            - condition:
                and:
                  - contains:
                      kubernetes.container.name: swift-client
                  - contains:
                      kubernetes.labels.app: swift-client
              config:
                - type: container
                  tail_files: true
                  tags: ["nextjs-server"]
                  pipeline: "nextjs-server"
                  multiline.pattern: 'WARN|ERROR|INFO|DEBUG'
                  multiline.negate: true
                  multiline.match: after
                  containers:
                    ids:
                      - ${data.kubernetes.container.id}
            - condition:
                and:
                  - contains:
                      kubernetes.container.name: istio-proxy
                  - contains:
                      kubernetes.labels.app: swift-client
                  - contains:
                      kubernetes.labels.app: swift-server
              config:
                - module: envoyproxy
                  access:
                    input:
                      type: container
                      tail_files: true
                      containers:
                        stream: stdout
                        ids:
                          - ${data.kubernets.container.id}
                  error:
                    input:
                      type: container
                      tail_files: true
                      containers:
                        stream: stderr
                        ids:
                          - ${data.kubernetes.container.id}

    processors:
      - add_cloud_metadata: ~
      - add_host_metadata: ~
      - add_kubernetes_metadata:
          in_cluster: true
      - add_docker_metadata:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']

I am new to filebeat and any help would be much appreciated! Thanks!

[faec (Fae Charlton)]

Welcome! I think the first error and possibly the second as well comes from your envoyproxy config, where you've put the input field underneath the access field. I believe input should be directly under the module definition, so instead of:

- module: envoyproxy
  access:
    input:
      type: container
      ...

try:

- module: envoyproxy
  input:
    type: container
    ...

[dnmahendra (Mahendra)]

Hi, Thank you so much that fixes those errors. Now, I have changed my config like this -

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    setup.dashboards.enabled: true
    setup.template.enabled: true
    setup.kibana.host: "192.168.82.162:5601"

    setup.template.settings:
      index.number_of_shards: 1
    setup.template.overwrite: true
    filebeat.overwrite_pipelines: true

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          templates:
            - condition.and:
                - contains:
                    kubernetes.container.name: istio-proxy
                - contains:
                    kubernetes.labels.app: swift-client
                - contains:
                    kubernetes.labels.app: swift-server
              config:
                input:
                  type: container
                  tail_files: true
                  containers:
                    stream: stdout
                    ids:
                      - ${data.kubernets.container.id}
            - condition.and:
                - contains:
                    kubernetes.container.name: swift-client
                - contains:
                    kubernetes.labels.app: swift-client
              config:
                input:
                  type: container
                  tail_files: true
                  tags: ["nextjs-server"]
                  pipeline: "nextjs-server"
                  multiline.pattern: 'WARN|ERROR|INFO|DEBUG'
                  multiline.negate: true
                  multiline.match: after
                  containers:
                    ids:
                      - ${data.container.id}

    processors:
      - add_cloud_metadata: ~
      - add_kubernetes_metadata:
          in_cluster: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']

I am using this - https://github.com/elastic/beats/tree/master/deploy/kubernetes/filebeat as reference to write the config. Is there any place I get some detailed docs on how to write this config. or maybe some working examples. Also, are these examples still valid - https://github.com/elastic/examples/blob/master/MonitoringKubernetes/filebeat-kubernetes.yaml ?

I am still seeing the dashboards broken on Kibana, not sure what changes to config I need to make to make the envoyproxy dashboard work on Kibana?

[faec (Fae Charlton)]

I took another look at this and it seems like a bug in the module; I've filed an issue in the project.