Filebeat Docker Autodiscovery stopped working when I upgraded to 8.12.0

Elastic Stack Beats
,
1

Hey,

Last week I was trying to use the logs from my Docker Swarm containers that were supposed to be in my test elasticsearch stack, when I found that said logs were not present. A bit of digging later and I can see that they stopped showing up the same day I upgraded the stack to 8.12.0.

As far as I can tell, I have Filebeat configured exactly the way the docs say.

And prior to 8.12.0, that config was working.

filebeat:
  autodiscover:
    providers:
      - type: docker
        hints.enabled: true
        host: "unix:///var/run/docker.sock"

With labels on the services like:

      co.elastic.logs/enabled: "true"
      co.elastic.logs/module: "apache"
      co.elastic.logs/fileset.stdout: "access"
      co.elastic.logs/fileset.stderr: "error"
      co.elastic.logs/processors.1.add_tags.tags: "drupal-logs"

Today I tried adding a default config template, but it did not help.

filebeat:
  autodiscover:
    providers:
      - type: docker
        hints.enabled: true
        host: "unix:///var/run/docker.sock"
        hints.default_config:
          type: container
          paths:
            - /var/lib/docker/containers/${data.container.id}/*.json-log

I can see filebeat registering the container log files, but it doesn't seem to be pushing the data to Elasticsearch.

The only clues I found were messages like this in filebeat's logs:

Feb 12 10:44:03 node02 filebeat[811900]: {"log.level":"warn","@timestamp":"2024-02-12T10:44:03.008-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'","service.name":"filebeat","ecs.version":"1.6.0"}

That is suspicious because I have nothing related to Kubernetes around. So why is Filebeat logging a Kube related error?

I did poke into the beats repo and found this: Use filestream input as default for hints autodiscover. (#36950) · elastic/beats@41ab08c · GitHub Which looks like it changed the defaults for autodiscovery to be aimed at Kubernetes. Which leaves me wondering where the code detecting that it should be using Docker config and not Kube is... Since I didn't find any where I was looking. (Not that I know golang beyond just knowing how to program in other languages.)

Anyway, any help would be appreciated.

Thanks!

2

Anyone? I'm still stuck on this.

3

What happens when you deploy exactly this (do not fix / change to your indenting style)

This is the reference config... problem though will probably be your file extension *.json-log

When you say it stopped working, what version did you upgrade from?

Can you share the rest of your config .yml? Is there some other provider?

There should be some other errors... do you see the connection to Elasticsearch?

You can set logging level to debug as well.

And why are you adding this here?

I don't think that is needed/correct if anything, that should be a processor

But is interesting looks like the default here ( i am not a go developer either)

processors:
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"

you could also try the default from the docs

filebeat.autodiscover.providers:
  - type: docker
    hints.enabled: true
    hints.default_config:
      type: container
      paths:
        - /var/lib/docker/containers/${data.container.id}/*.json-log
4

I was on 8.11.n prior to the 8.12.0 upgrade that broke autodiscovery.

Filebeat is able to send non-docker container logs to Elasticsearch just fine. So the ES connection is good.

I tried debug logs and didn't spot anything helpful at the time. I'll try again.

I set the autodiscovery host host: "unix:///var/run/docker.sock" to the default value just to see if it helped. It didn't.

I have tried the "default_config" and it didn't work.

I just tried adding the processors config, and it did not help.

Here's my config:

processors:
  - add_cloud_metadata: ~
filebeat:
  autodiscover:
    providers:
    - type: docker
      hints.enabled: true
        #    - hints.default_config:
        #paths:
        #- /var/lib/docker/containers/${data.container.id}/*.json-log
        #type: container
        #hints.enabled: true
        #host: unix:///var/run/docker.sock
        #type: docker
  config:
    inputs:
      enabled: true
      path: inputs.d/*.yml
      reload.enabled: true
      reload.period: 30s
    modules:
      enabled: true
      path: modules.d/*.yml
      reload.enabled: true
      reload.period: 30s
  modules:
  - audit:
      enabled: false
    auth:
      enabled: true
      var.paths:
      - /host/var/log/auth*
      - /host/var/log/secure*
    module: system
    syslog:
      enabled: true
      var.paths:
      - /host/var/log/messages*
      - /host/var/log/syslog*
output:
  elasticsearch:
    enabled: true
    hosts:
    - https://< user/pass/host >:9200
    index: projectname-logs-%{[agent.version]}
    ssl:
      certificate_authorities:
      - /usr/local/share/ca-certificates/projectname-ca.crt
      enabled: true
      verification_mode: certificate
setup:
  dashboards:
    enabled: false
  ilm:
    enabled: false
  json:
    data_stream: true
    enabled: false
  kibana:
    enabled: false
  template:
    enabled: false
    name: projectname-logs-%{[agent.version]}
    pattern: projectname-logs-%{[agent.version]}-*

And journald spit out these logs for filebeat:

Feb 20 09:59:26 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:26.998-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'log'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.012-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'log'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.012-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.013-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.013-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'error'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.014-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'error'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.015-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'access'","service.name":"filebeat","ecs.version":"1.6.0"}

I also tried adding:

  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"

to the processors and it did not help. Still get the missing kube field message.

I eventually tried a bunch of things and stripped the config down to:

log.level: debug
processors:
  - add_docker_metadata: ~
filebeat:
  autodiscover:
    providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/lib/docker/containers/${data.container.id}/*json.log
output:
  elasticsearch:
    enabled: true
    hosts:
    - https://< user/pass/host >:9200
    index: projectname-logs-%{[agent.version]}
    ssl:
      certificate_authorities:
      - /usr/local/share/ca-certificates/projectname-ca.crt
      enabled: true
      verification_mode: certificate
setup:
  dashboards:
    enabled: false
  ilm:
    enabled: false
  json:
    data_stream: true
    enabled: false
  kibana:
    enabled: false
  template:
    enabled: false
    name: projectname-logs-%{[agent.version]}
    pattern: projectname-logs-%{[agent.version]}-*

And I see logs like:

Feb 20 11:11:36 swarmnode02 filebeat[4053811]: {"log.level":"error","@timestamp":"2024-02-20T11:11:36.713-0800","log.logger":"reader_json","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*JSONReader).decode","file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character '-' after array element","service.name":"filebeat","ecs.version":"1.6.0"}

None of the search results I found about json decoding issues helped, and I'm pretty sure it's a red herring.

So, yeah, I'm running in circles at this point.

According to the docs: Hints based autodiscover | Filebeat Reference [8.12] | Elastic I should be able to just do:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false

I just tried that (again, I think...) and did not get anything new.

That, plus the logs about the missing kube field make it clear that Filebeat is trying to configure the access/error logs for some of the containers I have the hint labels on. But instead of looking for docker fields, it's looking for kube fields.

It really feels like a bug to me. Especially since my original config was working prior to the upgrade.

My old config:

filebeat:
  autodiscover:
    providers:
    - type: docker
      hints.enabled: true
      host: "unix:///var/run/docker.sock"

Any other ideas?

5

Nope... Pinged internally...

6

Thanks.

7

Did you try putting that in quotes?

8

We have nearly exact the same problem, just the error/warnings are different for us.

autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'

autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths' (source:'/etc/filebeat.yml')

autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'

Our config looks like this

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
            - config:
              - type: container
                paths:
                  - "/var/log/containers/*${data.kubernetes.container.id}.log"
                multiline.pattern: '^[[:space:]]'
                multiline.negate: false
                multiline.match: after

We used filebeat 8.11.4 before and everything worked fine, now everything seems to work too but just these messages are spammed into our log system.

9

@jerrac You should use the hints.default_config like

hints.default_config:
      type: container
      paths:
        - /var/lib/docker/containers/${data.container.id}/*.log

This will override the default config of filestream that was added in 8.12.
Is the log path you are using /var/lib/docker/containers/${data.container.id}/*json.log or /var/lib/docker/containers/${data.container.id}/*.json-log because the seconds seems wrong ?

10

@GEownt what are you trying to do with this configuration?
When hints are used along with templates, then hints will be evaluated only in case there is no template’s condition that resolves to true.
In your configuration there is a template with no condition, so it will be always true and hints won't be evaluated.
Are you running filebeat inside a kubernetes cluster?
Then why not use

filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
           paths:
           - /var/log/containers/*-${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           - multiline:
              type: pattern
              pattern: '^[[:space:]]'
              negate: false
              match: after
           prospector:
            scanner:
              fingerprint.enabled: true
              symlinks: true
           file_identity.fingerprint: ~

Can you try that?

11

Log path is *-json.log, I had mixed it up previously. Though I still am not getting the logs.

filebeat:
  autodiscover:
    providers:
    - hints.default_config:
        paths:
        - /var/lib/docker/containers/${data.container.id}/*.log
        type: container
      hints.enabled: true
      type: docker

Seems to have done the trick. Thanks.

I will note that that config will pull in logs from ALL containers. Not just the ones you have labeled with the hints.

Will there be a fix to restore the previous behavior?

12

That was actually buried in there... that is why I asked if you had surrounded with quotes..

Glad you found it

Not yaml professor but when it is a list and there is a - in the content... think is should be surrounded in quotes..

13

I am trying this configuration but then I get plenty of this messages

{"log.level":"warn","@timestamp":"2024-02-22T08:12:07.984Z","log.logger":"scanner","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles","file.name":"filestream/fswatch.go","file.line":389},"message":"cannot create a file descriptor for an ingest target \"/var/log/containers/calico-node-cz787_calico-system_flexvol-driver-4985acc5f36d15ed5da68fb6795955deef5d06dfbd2a10560135ff8b42597a2d.log\": filesize of \"/var/log/containers/calico-node-cz787_calico-system_flexvol-driver-4985acc5f36d15ed5da68fb6795955deef5d06dfbd2a10560135ff8b42597a2d.log\" is 85 bytes, expected at least 1024 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}

and this one still exists

autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id' (source:'/etc/filebeat.yml')

I am using filebeat in my kubernetes cluster.

14

Glad that it is working for you. The previous behaviour won't be restored as the update from container input to filestream was the intention.
But there should be a fix so that the default_config does not look for kubernetes variables in the events in case of docker provider.

15

For the first one, it is just a warning that some files are too small in size for filebeat to monitor them. This is due to the fingerprint mode

For the second error I will try to reproduce it. Meanwhile can you remove the data.kubernetes.container.id part from the id in the config block?
like:
id: kubernetes-container-logs-${data.kubernetes.pod.name}

16

The error messages are now gone. Is there any way to suspend the fingerprint warning messages? Because they are nearly flooding my logs.

17

You can either disable the fingerprint feature by removing fingerprint.enabled and file_identity.fingerprint: ~ which is not recommended , either configure it to reduce the fingerprint.length like

file_identity.fingerprint:
  enabled: false
  offset: 0
  length: 64
18

That's really what I meant, I just worded it badly. :smiley:

Thanks for the help!

2 Likes
19

Thanks for your help, now everything works as expected.
I just get a few warning messages now where the filesize is 0 bytes but thats ok because I can filter them out.

Here is my configuration for reference:

    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}
           paths:
           - /var/log/containers/*${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           - multiline:
              type: pattern
              pattern: '^[[:space:]]'
              negate: false
              match: after
           prospector:
            scanner:
              symlinks: true
              fingerprint:
                enabled: true
                offset: 0
                length: 64
           file_identity.fingerprint: ~
1 Like
20

I can share some findings with you. The warning message you were receiving autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths' is a false positive, meaning everything works as expected anyway.
This was initially a warning message that could happen when unpacking a config while applying the autodiscover config template. You can read it here.
In 8.12.2 this message logging level was brought down to debug so it is not there anymore.
I would suggest you upgrade to 8.12.2 and set your config like this

filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
           paths:
           - /var/log/containers/*${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           - multiline:
              type: pattern
              pattern: '^[[:space:]]'
              negate: false
              match: after
           prospector:
            scanner:
              symlinks: true
              fingerprint:
                enabled: true
                offset: 0
                length: 64
           file_identity.fingerprint: ~

as you will avoid any chance of data duplication in cases where there are pods with same name. Their container.id though can never be the same, which makes the id field of the config really unique.

1 Like