Filebeat - Kubernetes Autodiscovery with ElasticSearch Module

[Jonathan_Mabrito]

Hi There,

Hoping someone might be able to help with getting Filebeats Autodiscovery up. I am trying to detect my Elasticsearch containers in my Kubernetes Cluster (using vanilla K8's).

Here is my proposed K8's file:

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: elasticstack

  name: filebeat-configmap

  labels:

    app: elasticsearch

    role: filebeat

data:

  filebeat.yml: |  

    filebeat.autodiscover:

      providers:

        - type: kubernetes

          node: ${NODE_NAME}

          appenders:

            - condition.contains:

                kubernetes.labels.app: elasticsearch

              config:

                - module: elasticsearch

                  server:

                    enabled: true

                    var.paths:

                      - /var/log/elasticsearch/*.log

                      - /var/log/elasticsearch/*_server.json

                  gc:

                    enabled: true

                    # Set custom paths for the log files. If left empty,

                    # Filebeat will choose the paths depending on your OS.

                    var.paths:

                      - /var/log/elasticsearch/gc.log.[0-9]*

                      - /var/log/elasticsearch/gc.log

                  audit:

                    enabled: true

                    # Set custom paths for the log files. If left empty,

                    # Filebeat will choose the paths depending on your OS.

                    var.paths:

                      - /var/log/elasticsearch/*_access.log

                      - /var/log/elasticsearch/*_audit.json

                  slowlog:

                    enabled: true

                    # Set custom paths for the log files. If left empty,

                    # Filebeat will choose the paths depending on your OS.

                    var.paths:

                      - /var/log/elasticsearch/*_index_search_slowlog.log

                      - /var/log/elasticsearch/*_index_indexing_slowlog.log

                      - /var/log/elasticsearch/*_index_search_slowlog.json

                      - /var/log/elasticsearch/*_index_indexing_slowlog.json

                  deprecation:

                    enabled: true

                    # Set custom paths for the log files. If left empty,

                    # Filebeat will choose the paths depending on your OS.

                    var.paths:

                      - /var/log/elasticsearch/*_deprecation.log

                      - /var/log/elasticsearch/*_deprecation.json

    processors:

      - add_cloud_metadata:

      - add_host_metadata:

    output.elasticsearch:

      hosts: ["http://elasticsearch-ingest.elasticstack.svc.cluster.local:9200"]

    setup.dashboards.enabled: true

    setup.template.enabled: true

    setup.kibana:

      host: kibana.elasticstack.svc.cluster.local:5601

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: filebeat

  namespace: elasticstack

  labels:

    app: elasticsearch

    role: filebeat

spec:

  selector:

    matchLabels:

      app: elasticsearch

      role: filebeat

  template:

    metadata:

      labels:

        app: elasticsearch

        role: filebeat

    spec:

      serviceAccountName: filebeat-svc

      terminationGracePeriodSeconds: 30

      hostNetwork: true

      dnsPolicy: ClusterFirstWithHostNet

      containers:

      - name: filebeat

        image: docker.elastic.co/beats/filebeat:7.17.1

        args: [

          "-c", "/etc/filebeat.yml",

          "-e",

        ]

        env:

        - name: NODE_NAME

          valueFrom:

            fieldRef:

              fieldPath: spec.nodeName

        securityContext:

          runAsUser: 0

          # If using Red Hat OpenShift uncomment this:

          #privileged: true

        resources:

          limits:

            memory: 200Mi

          requests:

            cpu: 100m

            memory: 100Mi

        volumeMounts:

        - name: config

          mountPath: /etc/filebeat.yml

          readOnly: true

          subPath: filebeat.yml

        - name: data

          mountPath: /usr/share/filebeat/data

        - name: varlibdockercontainers

          mountPath: /var/lib/docker/containers

          readOnly: true

        - name: varlog

          mountPath: /var/log

          readOnly: true

      volumes:

      - name: config

        configMap:

          defaultMode: 0640

          name: filebeat-configmap

      - name: varlibdockercontainers

        hostPath:

          path: /var/lib/docker/containers

      - name: varlog

        hostPath:

          path: /var/log

      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart

      - name: data

        hostPath:

          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).

          path: /var/lib/filebeat-data

          type: DirectoryOrCreate

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: filebeat-clsbind

subjects:

- kind: ServiceAccount

  name: filebeat-svc

  namespace: elasticstack

roleRef:

  kind: ClusterRole

  name: filebeat-clsrole

  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: filebeat-clsrole

  labels:

    app: elasticsearch

    role: filebeat

rules:

- apiGroups: [""] # "" indicates the core API group

  resources:

  - namespaces

  - pods

  verbs:

  - get

  - watch

  - list

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: filebeat-svc

  namespace: elasticstack

  labels:

    app: elasticsearch

    role: filebeat

The Daemonset comes online, but I do not see anything show up in Kibana or the Filebeats index. I am pretty sure the Elasticsearch containers are not being discovered and not entirely sure why. Any ideas of whats defined wrong in this configuration?