Filebeat for Kubernetes

Im using Kubernetes and would like to add filebeat to all stdouts.
I have followed: https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml and have the filebeat running.
How ever it is not able to access the password as an environmet variable.

I have tried mounting it with both a secret and as a raw value.

With output.elasticsearch.password: ${ELASTICSEARCH_PASSWORD} i get error:
ERROR instance/beat.go:743 Exiting: error initializing publisher: can not convert 'object' into 'string' accessing 'output.elasticsearch.password' (source:'/etc/filebeat.yml')

Best

I've seen this problem outside of kubernetes, can you try starting Filebeat locally with the same config and same password set in the environment. I do not see your configuration, but the error above was generally due to a malformed yaml file.

I did try just replacing ${ELASTICSEACH_PASSWORD} with the password, no "", and that worked perfectly :confused:
Also tried raw text in env, thinking it was a secret related problem but still same error.

I have not tried locally but will do that durring the day.
In the mean time the daemon and input-config:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: commoninfra-admin
  labels:
    k8s-app: filebeat
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 5
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.4.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_USERNAME
          value: aks-staging-filebeat
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: aks-filebeat
              key: password
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: dockersock
          mountPath: /var/run/docker.sock
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate


apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: commoninfra-admin
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    logging.level: debug
    setup:
      template:
        name: "filebeat"
        pattern: "filebeat-*"
        enabled: false
    filebeat.autodiscover:
      providers:
        - type: docker
          templates:
            - condition:
                equals:
                  docker.container.labels.io.kubernetes.container.name: "nginx-ingress-controller"
              config:
                - module: nginx
                  access:
                    input:
                      type: docker
                      containers.stream: stdout
                      containers.ids:
                        - "${data.docker.container.id}"
                  error:
                    input:
                      type: docker
                      containers.stream: stderr
                      containers.ids:
                        - "${data.docker.container.id}"
    processors:
      - drop_event:
          when.not:
            or:
              - contains:
                  docker.container.labels.io.kubernetes.container.name: "nginx-ingress-controller"
    output:
      elasticsearch:
        hosts: ["my_hostname_here"]
        username: aks-staging-filebeat
        password: ${ELASTICSEARCH_PASSWORD}
        index: "filebeat-aks-staging-%{+yyyy.MM.dd}"
        ssl.verification_mode: "none"

I solved it.
After reissuing the secret with a UTF8 encoding of the base64 encoded string it started working.
Don't know what standard mac base64 encode is using but it might be worthy investigate that format since that works with other applications in kubernetes.

Best regards
Andreas

Thanks @tehho I will try to replicate that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.