Filebeat 7.13.4 + Kubernetes 1.20: Hints-Based Autodiscover

Elastic Stack Beats
1

We just upgraded our Kubernetes cluster to 1.20 (current version of filebeat is 7.13.4). Of course, this means that Kubernetes logs are no longer in the same place they were under 1.19, and instead are in the following location:

/var/logs/containers/*-<container_id>.log

We use hints-based autodiscover in our cluster, with annotations only; there is no default defined.

Since my previous autodiscover rule is no longer working:

- type: 'kubernetes'
  hints:
    enabled: true
    default_config:
      enabled: false
  processors:
    - add_kubernetes_metadata: ~

I've updated my autodiscover provider rule to the following:

- type: 'kubernetes'
  combine_partial: true
  cri:
    force: true
    parse_flags: true
  hints:
    enabled: true
    default_config:
      enabled: false
  paths:
    - '/var/log/containers/*-${data.container.id}.log'
  processors:
    - add_kubernetes_metadata: ~

However, my logs aren't ingesting as expected. I'm certain that I just have my provider misconfigured, but I've tried several variations, without any success. When I run filebeat in debug mode, it seems to locate the annotated containers, but the rules don't load.

Edit:

No matter what I do for the config, it attempts to load the dockerd paths:

2021-10-12T01:34:06.647Z    INFO    log/input.go:157    Configured paths: [/var/lib/docker/containers/3716ebf97e0353a0f7ac9a0328c0142d87ee8370c25669846b4914e34ba372bd/*-json.log]

Edit #2:

After a deeper search, I found this question regarding the same topic:

Filebeat autodiscover for Kubernetes uses inconsistent log files path by default

I'm doing an additional dive to determine whether that issue has been resolved yet.

Update: I haven't been able to find any indication that this has been resolved (or even that it was raised as an issue). Anybody know if it has?

Thanks.

2

I've validated that this still appears to be the case in 7.15.0. I was unable to find a ticket opened by the user in the linked-to issue, so I've opened one.

3

Under 7.15.0, specifying the path under default_config forces the use of the correct path:

- type: 'kubernetes'
  hints:
    enabled: true
    default_config:
      enabled: false
      type: 'container'
      paths:
        - '/var/log/containers/*-${data.container.id}.log'
      cri:
        force: true
        parse_flags: true
  processors:
    - add_kubernetes_metadata: ~
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.