Filebeat Autodiscover on Kubernetes not working

Paul_B · February 12, 2024, 6:00pm

I'm having trouble getting autodiscover to work correctly in a Kubernetes environment, if I restart the Filebeat daemonset then logs of new pods are collected correctly but if a pod restarts then Filebeat doesn't see the change.

My configuration for Filebeat looks like this:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints:
        enabled: true
        default_config:
          type: container
          paths:
            - /var/log/containers/*${data.container.id}.log

stephenb · February 13, 2024, 3:42pm

Hi @Paul_B

1st autodiscover can be hard to debug.

and we are going to need a lot more detail if you would like help

What version of filebeat are you on?

Is that the whole autodiscover section? Is there anything below it?

What hints are you using? Do you mean to use hints?

Can you share the actual annotations you have added?

What are you trying to accomplish? / What is your expected result?

What IS working right .. meaning did you apply hints and then get the result you want?

What change are you making to the annotation / hints or just restarting?

Pretty sure you can see the whole kubernets object by just adding a field

           fields:

              # You can see the whole kubernetes object... 
              fields.kubernetes: "${kubernetes}"

Paul_B · February 14, 2024, 8:32am

Thanks for getting back to me @stephenb , to answer your questions:

What change are you making to the annotation / hints or just restarting?

Sorry for not being clear, when I say "change" I mean Filebeat is harvesting the logs of a Kubernetes Pod just fine, however, if that Pod is deleted (manually or via an upgrade) and a new Pod is created, the logs from the new Pod no longer get sent to Elastic.

To show an example this dashboard is focussed on a single backend micro-service which was upgraded at 1400, when we looked at the logs the next morning we saw they were missing. We then had to manually restart the Filebeat daemonset which then start harvesting the logs of the new Pod:

What are you trying to accomplish? / What is your expected result?
What IS working right .. meaning did you apply hints and then get the result you want?

We've been using Filebeat Autodiscover since around the latest version 7.x.x; around 3 months ago we upgraded our Elastic Cloud instance to latest version 8 leaving Filebeat running 7. We more recently upgraded Filebeat to 8.12.0 at the time of writing this and we've noticed that we're losing logs on Pod restarts.

Can you share the actual annotations you have added?

Sure, all the Pods have the following annotations that single out only picking up logs for the "*-service" container in the Pod:

Is that the whole autodiscover section? Is there anything below it?

The full autodiscover section is shown below:

Summary

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.container.id}.log
      templates:
        # Redis Module
        - condition:
            contains:
              kubernetes.container.image: redis
          config:
            - module: redis
              log:
                input:
                  type: container
                  paths:
                    - '/var/log/containers/*-${data.kubernetes.container.id}.log'
                  pipeline: ${Ingest_Pipeline_Redis_Log:INGEST_PIPELINE_REDIS_LOG_NOT_SET}
              slowlog:
                enabled: true
                input:
                  pipeline: ${Ingest_Pipeline_Redis_SlowLog:INGEST_PIPELINE_REDIS_SLOWLOG_NOT_SET}
                var.hosts: ["${data.host}:${data.port}"]

What version of filebeat are you on?

We're using 8.12.0

I think the main issue is the ambiguity in the documentation, for example, when it says

You can configure the default config that will be launched when a new container is seen, like this:

And shows the following code:

It follows up with:

You can also disable default settings entirely, so only Pods annotated like co.elastic.logs/enabled: true will be retrieved:

And shows:

Does this mean we don't need the "paths" bit any more? We definitely only want the logs annotated with "co.elastic.logs/enabled: true".

In fact, is that the problem? Should we just drop the paths property and use "hints.default_config.enabled: false"?

stephenb · February 14, 2024, 6:56pm

Hi @Paul_B

First, in the future, can you paste text, not screenshots of text much harder to work with...

Ok here is what I did...

I went and downloaded the filebeat for Kubernetes reference found here

This is always where I start from, or take a look if upgrading. This is the reference.

I updated to use the autodiscover out of the box....
Note: There is little change on the creds as mine are in secrets that I convert to ENV Vars
Note: I have no hints / annotations on any apps. (back to that later)

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # filebeat.inputs:
    # - type: filestream
    #   paths:
    #     - /var/log/containers/*.log
    #   parsers:
    #     - container: ~
    #   prospector:
    #     scanner:
    #       fingerprint.enabled: true
    #       symlinks: true
    #   file_identity.fingerprint: ~
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
           paths:
           - /var/log/containers/*-${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           prospector:
            scanner:
              fingerprint.enabled: true
              symlinks: true
           file_identity.fingerprint: ~

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    # cloud.id: ${ELASTIC_CLOUD_ID}
    # cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ES_HOST}']
      username: ${ES_USERNAME}
      password: ${ES_PASSWORD}
---

This works as expected it picks up all my apps, and when I kill them or restart them or delete them or redploy them logs continue to flow.

So I would start with that and then see if your annotations work...

Which speaking of ... I am not sure exactly how you are pulling those annotations because you did not show the command. My annotations do not have the app name in them

These annotations are for heartbeat autodiscover

hyperion:k8s sbrown$ kubectl get service my-otel-demo-frontend   -o jsonpath='{.metadata.annotations}' | jq
{
  "cloud.google.com/neg": "{\"ingress\":true,\"exposed_ports\":{\"8080\":{}}}",
  "cloud.google.com/neg-status": "{\"network_endpoint_groups\":{\"8080\":\"k8s1-f6c1ea2c-default-my-otel-demo-frontend-8080-8604104a\"},\"zones\":[\"us-west2-a\",\"us-west2-b\",\"us-west2-c\"]}",
  "co.elastic.monitor/hosts": "my-otel-demo-frontend.default.svc.cluster.local:8080",
  "co.elastic.monitor/id": "my-otel-demo-frontend-monitor-id-1234",
  "co.elastic.monitor/name": "my-otel-demo-frontend",
  "co.elastic.monitor/schedule": "@every 10s",
  "co.elastic.monitor/timeout": "5s",
  "co.elastic.monitor/type": "http",
  "meta.helm.sh/release-name": "my-otel-demo",
  "meta.helm.sh/release-namespace": "default"
}

For filebeat the annotation should look as shown here

Not sure what yours are showing the service name in the path.

Paul_B · February 14, 2024, 8:04pm

Thanks for the response @stephenb .

I've annotated the pods according to the documentation for pods with multiple containers as we have a number of sidecars alongside our software services and for the time being we're not interested in their logs:

Multiple containersedit
When a pod has multiple containers, the settings are shared unless you put the container name in the hint. For example, these hints configure multiline settings for all containers in the pod, but set a specific exclude_lines hint for the container called sidecar.

annotations:
  co.elastic.logs/multiline.pattern: '^\['
  co.elastic.logs/multiline.negate: true
  co.elastic.logs/multiline.match: after
  co.elastic.logs.sidecar/exclude_lines: '^DBG'

I fear I may have wasted your time a bit here, I did what you suggested and took a look at the reference file. Comparing it to what I posted earlier I ended up removing the "type" and "paths", replacing that block with

hints.default_config.enabled: false

After doing this I noticed errors in the filebeat logs mentioning the Redis Module inputs so I removed that entire block as well, restarted Filebeat and the autodiscover is now working

My final Filebeat config is now simply:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config.enabled: false

Many thanks for the help and for pointing me in the right direction

system · March 13, 2024, 10:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeats autodiscover on kubernetes doesn't collect logs from annotated pods Beats filebeat	1	323	June 2, 2022
Autodiscover not working Beats filebeat	2	326	July 6, 2023
Kubernetes autodiscover with annotation not working Beats filebeat	1	309	September 3, 2021
Filebeat 7.13.4 + Kubernetes 1.20: Hints-Based Autodiscover Beats filebeat	3	567	November 10, 2021
Filebeat autodiscover not working with hints & namespace Beats filebeat	3	2099	September 15, 2020

Filebeat Autodiscover on Kubernetes not working

Related topics