Trouble with configuring Filebeat as DaemonSet on k8s environment

cloudants_wvestjens · January 29, 2020, 3:38pm

Hello.

I am trying to configure Filebeat as DaemonSet on our Kubernetes platform.

It's sending through systemlogs as expected, including the event info.
I am trying to get it to do the same for nginx, apache2 and eventually other modules.
However, there is no information / logs coming in (any more?) for apache / nginx containers.

These are my configs:

filebeat-daemonset.yml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
    name: filebeat
    namespace: {{ .Values.namespace }}
    labels:
        k8s-app: filebeat
spec:
    selector:
        matchLabels:
            k8s-app: filebeat
    template:
        metadata:
            labels:
                k8s-app: filebeat
        spec:
            serviceAccountName: filebeat
            terminationGracePeriodSeconds: 30
            hostNetwork: true
            dnsPolicy: ClusterFirstWithHostNet
            imagePullSecrets:
              - name: gitlab-registry
            containers:
                - name: filebeat
                  image: {{ .Values.filebeat.image }}
                  imagePullPolicy: Always
                  args: ["-e"]
                  env:
                      - name: ELASTICSEARCH_HOST
                        value: "{{ .Values.elasticsearch.host }}"
                      - name: ELASTICSEARCH_PORT
                        value: "{{ .Values.elasticsearch.port }}"
                      - name: ELASTICSEARCH_USERNAME
                        value: "{{ .Values.elasticsearch.user }}"
                      - name: ELASTICSEARCH_PASSWORD
                        value: "{{ .Values.elasticsearch.password }}"
                      - name: KIBANA_HOST
                        value: "{{ .Values.kibana.host }}"
                      - name: KIBANA_PORT
                        value: "{{ .Values.kibana.port }}"
                      - name: KIBANA_PROTOCOL
                        value: "{{ .Values.kibana.protocol }}"
                      - name: KIBANA_PATH
                        value: "{{ .Values.kibana.path }}"
                      - name: KIBANA_USER
                        value: "{{ .Values.kibana.user }}"
                      - name: KIBANA_PASSWORD
                        value: "{{ .Values.kibana.password }}"
                      - name: NODE_NAME
                        valueFrom:
                            fieldRef:
                                fieldPath: spec.nodeName
                  securityContext:
                      runAsUser: 0
                  resources:
                      limits:
                          memory: 200Mi
                      requests:
                          cpu: 100m
                          memory: 100Mi
                  volumeMounts:
                      - name: data
                        mountPath: /usr/share/filebeat/data
                      - name: varlibdockercontainers
                        mountPath: /var/lib/docker/containers
                        readOnly: true
                      - name: varlog
                        mountPath: /var/log
                        readOnly: true
            volumes:
                - name: varlibdockercontainers
                  hostPath:
                      path: /var/lib/docker/containers
                - name: varlog
                  hostPath:
                      path: /var/log
                - name: data
                  hostPath:
                      path: /var/lib/filebeat-data
                      type: DirectoryOrCreate

filebeat.yml:

setup.dashboards.enabled: true
setup.template.enabled: true
setup.template.settings:
  index.number_of_shards: 1

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - config:
          - module: system
            syslog:
              enabled: true
            auth:
              enabled: true
        - condition.contains:
            kubernetes.labels.app: redis
          config:
            - module: redis
              log:
                input:
                  type: docker
                  containers.ids:
                    - ${data.kubernetes.container.id}
              slowlog:
                enabled: true
                var.hosts: ["${data.host}:${data.port}"]
        - condition.contains:
            kubernetes.labels.app: apache
          config:
            - module: apache2
              access:
                input:
                  type: docker
                  containers.ids:
                    - ${data.kubernetes.container.id}
              error:
                input:
                  type: docker
                  containers.ids:
                    - ${data.kubernetes.container.id}
        - condition.equals:
            kubernetes.labels.app: mysql
          config:
            - module: mysql
              error:
                input:
                  type: docker
                  containers.ids:
                    - ${data.kubernetes.container.id}
              slowlog:
                input:
                  type: docker
                  containers.ids:
                    - ${data.kubernetes.container.id}

processors:
  - drop_event:
      when.or:
        - and:
            - regexp:
                message: '^\d+\.\d+\.\d+\.\d+ '
            - equals:
                fileset.name: error
        - and:
            - not:
                regexp:
                  message: '^\d+\.\d+\.\d+\.\d+ '
            - equals:
                fileset.name: access
  - add_cloud_metadata:
  - add_kubernetes_metadata:
  - add_docker_metadata:

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

setup.kibana:
  host: "${KIBANA_HOST:localhost}"
  port: "${KIBANA_PORT:5601}"
  protocol: "${KIBANA_PROTOCOL:https}"
  username: "${KIBANA_USER:elastic}"
  password: "${KIBANA_PASSWORD}"
  ssl:
    enabled: true

The Docker filebeat image is built from the following Dockerfile:

ARG FILEBEAT_VERSION
FROM docker.elastic.co/beats/filebeat:${FILEBEAT_VERSION}

# Elevated permissions are necessary to set proper permissions
USER 0
ADD --chown=root:filebeat config/filebeat.yml /usr/share/filebeat/
RUN chmod go-w /usr/share/filebeat/filebeat.yml
USER 1000

# Enable modules
RUN filebeat modules enable mysql nginx apache redis system elasticsearch haproxy iptables kibana logstash rabbitmq traefik

Any help would be appreciated, I'm a bit lost at the moment.

Kind regards,
Wesley

ChrsMark · January 30, 2020, 8:26am

Hi!

Could you try adding the path for these modules? So as the config to look like:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.image: "redis"
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - /var/log/container/*-${data.kubernetes.container.id}.log

From the docs.

Also do you see anything interesting in Filebeat's logs regarding autodiscover? Anything regarding the apache Module's configuration for instance?

If this does not help, you should dive deeper and debug the Fileebat by running manually the binary inside the Daemonset in debug level and see how autodiscover detects the containers/pods and with which configuration starts the modules?

cloudants_wvestjens · January 30, 2020, 2:36pm

Hi Chris,

Thank you for coming back to me. I've added the paths, if that doesn't help I'll reply again.

However, in the meantime, how do you enable the filebeat logs / where can I see them? Is it simply the kubectl logs output for the filebeat pods? I've not been able to detect any thing that hints to errors there, I can see it's looking for the apache information but it does not report on errors or anything that isn't found (as far as I can see).

Regards,
Wesley

cloudants_wvestjens · January 30, 2020, 2:42pm

Hi Chris,

I've looked at the results / what is pushed to ES right now, but it's only sending syslogs.
It's as if it's not recognising the module config at all.

It's also not really doing what we'd like it to do. If the log record matches one of the modules, it should be parsed as is configured in that module.

If it doesn't match any known module / syntax, we want the log record to be sent as-is, we don't want any log records to not be sent / excluded unless we explicitly configure it to be skipped...

cloudants_wvestjens · January 30, 2020, 3:47pm

Additionally, I've tried other configurations which are taken from the docs.
There are multiple ways reported to enable modules, I've activated 2 of them and neither seem to work:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - add_docker_metadata:
        - add_host_metadata:

    filebeat.modules:
    - module: nginx
    - module: mysql
    - module: system

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.5.2
          imagePullPolicy: Always
          args: [
            "-c", "/etc/filebeat.yml",
            "--modules", "nginx,apache,mysql,haproxy",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: "{{ .Values.elasticsearch.host }}"
            - name: ELASTICSEARCH_PORT
              value: "{{ .Values.elasticsearch.port }}"
            - name: ELASTICSEARCH_USERNAME
              value: "{{ .Values.elasticsearch.user }}"
            - name: ELASTICSEARCH_PASSWORD
              value: "{{ .Values.elasticsearch.password }}"
            - name: KIBANA_HOST
              value: "{{ .Values.kibana.host }}"
            - name: KIBANA_PORT
              value: "{{ .Values.kibana.port }}"
            - name: KIBANA_PROTOCOL
              value: "{{ .Values.kibana.protocol }}"
            - name: KIBANA_PATH
              value: "{{ .Values.kibana.path }}"
            - name: KIBANA_USER
              value: "{{ .Values.kibana.user }}"
            - name: KIBANA_PASSWORD
              value: "{{ .Values.kibana.password }}"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""] # "" indicates the core API group
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---

As you can see, I am specifying the --modules argument and I am specifying modules in the config, but no modules are enabled.

ChrsMark · January 31, 2020, 9:36am

Hi!

I think that the latest configuration you posted

    filebeat.modules:
    - module: nginx
    - module: mysql
    - module: system

will not work since modules will need to "know" the logs' paths like /var/log/containers/*${data.kubernetes.container.id}.log like in the autodiscover examples. The module by default will use /var/log/redis/redis-server.log*.

Since the setup you are trying to deploy is not so simple to work out of the box I would suggest to start by trying a simple setup with only one autodiscover condition and one service to be monitored like the one I provided to you in my previous post.

Along with this, you can kubectl exec -it filebeat-pod -- /bin/bash and start a second instance of filebeat so as to follow the logs like ./filebeat -e -d "*", and check in the logs if you see anything related to autodiscover and if the modules are detected and able to start.

C.

cloudants_wvestjens · January 31, 2020, 3:19pm

Hi Chris,

I am trying the debug option at the moment, however I think I still have errors in my configuration.

I am currently running on this config:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log
      condition.contains:
        - kubernetes.labels.app: nginx
      config:
        - module: nginx
          access:
            enabled: true
            var.paths:
              - /var/log/containers/*-${data.kubernetes.container.id}.log
          error:
            enabled: true
            var.paths:
              - /var/log/containers/*-${data.kubernetes.container.id}.log

processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"
  - add_docker_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

However I am receiving syslogs but no logs from nginx. Debug mode shows some log records that add kubernetes meta data, but nothing in relation to nginx logs (while I am reloading the frontend running on nginx).

The output of filebeat modules list also shows no modules enabled, but I am not sure what the proper way to activate them is. Specifying the --modules argument does not seem to activate the modules, building a separate image and running filebeat modules enable does.

I should note though, that I am not sure if the var.paths config I am currently using is correct, but I have some troubles finding out what the proper solution is (opinions seem to differ online).

That said, I do not really understand why I am receiving syslogs, but no logs from nginx, even though that's the only module I've configured...

Any guidance in getting a first module setup would be greatly appreciated.

P.s.: as extra information, this is an excerpt from the kubectl describe pod -n <NS> -o yaml <podname>:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx

Regards,
Wesley

ChrsMark · January 31, 2020, 3:34pm

Hi again!

Autodiscover is responsible to identify any incoming matching pods and spawn the respective Module to monitor them. So in this case the nginx pod should be caught by autodiscover watcher and Metricbeat would start an nginx module under the hood. All these could be tracked in Filebeat's logs.

In your configuration, I see some things like hints and stuff and I'm not sure how all these are mixed. I could only trust the minimal configuration example from the docs as the starting point for debugging:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.image: "redis"
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - /var/log/container/*-${data.kubernetes.container.id}.log

You can switch it to nginx accordingly.

Having said these, I would suggest to try with that and if nothing shows up in the logs (running Filebeat in debug mode `./filebeat -e -d "*') regarding autodiscover then it seems that something goes wrong with the conditions and the config cannot catch the Pod creation event. Please try to debug (feel free to share the logs with us) this in this structured way and if there is no success I will be able to try it on my end for you.

Regards, C.

system · February 28, 2020, 3:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.