Filebeat not shipping kubectl logs output

Elastic Stack Beats
1

Hi all. I am trying to ship kubectl logs output to Elasticsearch with Filebeat. Example command:

kubectl logs --tail=10 --timestamps ceo-container1 -n ceo-dev1

Example output:

2024-09-17T15:08:44.375437739Z 2024-09-17T15:08:44.375Z  INFO 1 --- [           main] com.intelsat.ceo.CeoAp    plication          : Started CeoApplication in 7.837 seconds (process running for 8.42)
2024-09-17T15:08:44.544767697Z On server start scheduler
2024-09-17T15:08:44.544825780Z No active schedule
2024-09-17T15:33:49.692192221Z 2024-09-17T15:33:49.691Z  INFO 1 --- [nio-8085-exec-1] o.a.c.c.C.[.[localhost    ].[/ceo/v1]        : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-09-17T15:33:49.692243653Z 2024-09-17T15:33:49.691Z  INFO 1 --- [nio-8085-exec-1] o.s.web.servlet.Dispat    cherServlet        : Initializing Servlet 'dispatcherServlet'
2024-09-17T15:33:49.694066083Z 2024-09-17T15:33:49.693Z  INFO 1 --- [nio-8085-exec-1] o.s.web.servlet.Dispat    cherServlet        : Completed initialization in 2 ms
2024-09-19T13:13:45.337897996Z 2024-09-19T13:13:45.337Z  INFO 1 --- [nio-8085-exec-6] o.springdoc.api.Abstra    ctOpenApiResource  : Init duration for springdoc-openapi is: 516 ms
2024-09-24T15:58:02.902999238Z Scheduling task with job id: 72 and cron expression: 0 */30 * * * ?
2024-09-24T16:00:00.001242789Z Running action: Test_Action
2024-09-24T16:00:00.001280873Z With Data: Test Data

I downloaded the manifest file at Run Filebeat on Kubernetes | Filebeat Reference [8.15] | Elastic, then made a copy of it. Then in the copied file I deleted the filestream input and uncommented the autodiscover section just as suggested in the comment. Thus the relevant section looks like this:

data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        hints.default_config:
          type: filestream
            id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
            paths:
            - /var/log/containers/*-${data.kubernetes.container.id}.log
            parsers:
            - container: ~
            prospector:
              scanner:
                fingerprint.enabled: true
              symlinks: true
            file_identity.fingerprint: ~

I then ran kubectl create -f on the copied file. However when I look at Discover in Kibana I don't see the data coming from the filebeat-* index pattern.

2

After discussion with the team, we decided to give up on autodiscover.

Instead the team will configure Spring Boot apps to log to explicitly named log files and Filebeat will collect from those log files via a filestream input.

3

Hi @paolovalladolid

I just deployed the latest to my "Stock GKE" cluster and it works as expected I only added the credentials...

I just deployed this and it started shipping the logs...

And to be clear when you say this.... you mean shipping container logs...

Did you look at the logs from the filebeat pods and look for errors?

I have seen in the past if you have many containers you might need to up the resources

        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:

What I deployed.....

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["apps"]
  resources:
    - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources:
    - jobs
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat
  # should be the namespace where filebeat is running
  namespace: kube-system
  labels:
    k8s-app: filebeat
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat-kubeadm-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: Role
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat-kubeadm-config
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: Role
  name: filebeat-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: filestream
            id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
            paths:
            - /var/log/containers/*-${data.kubernetes.container.id}.log
            parsers:
            - container: ~
            prospector:
              scanner:
                fingerprint.enabled: true
                symlinks: true
            file_identity.fingerprint: ~

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.15.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value: mydemo:lskjfhslkfjhljkhlkjhlkjh0MyRjOTU3YWMyZmJkMWY0YTAwYTIzM2U4YzgyMTcyZjc5YyRjZTU4YTljMGY5NGE0MjY2YjllNzBmZDA5OTZjODdjYg==
        - name: ELASTIC_CLOUD_AUTH
          value: elastic:skdjghsldkfgjhsdlfgkjh
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
4

Hi @stephenb.

Some of the lines in this example output were created by calls to System.out.println, like No active schedule and Running action: Test_Action

2024-09-17T15:08:44.375437739Z 2024-09-17T15:08:44.375Z  INFO 1 --- [           main] com.intelsat.ceo.CeoAp    plication          : Started CeoApplication in 7.837 seconds (process running for 8.42)
2024-09-17T15:08:44.544767697Z On server start scheduler
2024-09-17T15:08:44.544825780Z No active schedule
2024-09-17T15:33:49.692192221Z 2024-09-17T15:33:49.691Z  INFO 1 --- [nio-8085-exec-1] o.a.c.c.C.[.[localhost    ].[/ceo/v1]        : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-09-17T15:33:49.692243653Z 2024-09-17T15:33:49.691Z  INFO 1 --- [nio-8085-exec-1] o.s.web.servlet.Dispat    cherServlet        : Initializing Servlet 'dispatcherServlet'
2024-09-17T15:33:49.694066083Z 2024-09-17T15:33:49.693Z  INFO 1 --- [nio-8085-exec-1] o.s.web.servlet.Dispat    cherServlet        : Completed initialization in 2 ms
2024-09-19T13:13:45.337897996Z 2024-09-19T13:13:45.337Z  INFO 1 --- [nio-8085-exec-6] o.springdoc.api.Abstra    ctOpenApiResource  : Init duration for springdoc-openapi is: 516 ms
2024-09-24T15:58:02.902999238Z Scheduling task with job id: 72 and cron expression: 0 */30 * * * ?
2024-09-24T16:00:00.001242789Z Running action: Test_Action
2024-09-24T16:00:00.001280873Z With Data: Test Data

We had started down the path of just leaving the System.out.println calls in the application code (Spring Boot and Java) and trying to harvest the output with Filebeat using the autodiscover features.

But then the team decided to replace the System.out calls with Logback or something like it to generate log files on the file system, then use Filebeat filestream input to collect data from those log files.

5

Then you need to figure out where they are writing the log files path wise...

The default config will look for logs in the default locations...

So where are the logs being written too...