# Filebeat Docker Autodiscovery stopped working when I upgraded to 8.12.0

**URL:** <https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [February 12, 2024, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103 "2024-02-12T19:25:07Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [February 12, 2024, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/1 "2024-02-12T19:25:07Z")

</div>

Hey,

Last week I was trying to use the logs from my Docker Swarm containers that were supposed to be in my test elasticsearch stack, when I found that said logs were not present. A bit of digging later and I can see that they stopped showing up the same day I upgraded the stack to 8.12.0.

As far as I can tell, I have Filebeat configured exactly the way the [docs](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_docker_3) say.

And prior to 8.12.0, that config was working.

```yaml
filebeat:
  autodiscover:
    providers:
      - type: docker
        hints.enabled: true
        host: "unix:///var/run/docker.sock"

```

With labels on the services like:

```yaml
      co.elastic.logs/enabled: "true"
      co.elastic.logs/module: "apache"
      co.elastic.logs/fileset.stdout: "access"
      co.elastic.logs/fileset.stderr: "error"
      co.elastic.logs/processors.1.add_tags.tags: "drupal-logs"

```

Today I tried adding a default config template, but it did not help.

```yaml
filebeat:
  autodiscover:
    providers:
      - type: docker
        hints.enabled: true
        host: "unix:///var/run/docker.sock"
        hints.default_config:
          type: container
          paths:
            - /var/lib/docker/containers/${data.container.id}/*.json-log

```

I can see filebeat registering the container log files, but it doesn't seem to be pushing the data to Elasticsearch.

The only clues I found were messages like this in filebeat's logs:

```auto
Feb 12 10:44:03 node02 filebeat[811900]: {"log.level":"warn","@timestamp":"2024-02-12T10:44:03.008-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'","service.name":"filebeat","ecs.version":"1.6.0"}

```

That is suspicious because I have nothing related to Kubernetes around. So why is Filebeat logging a Kube related error?

I did poke into the beats repo and found this: [Use filestream input as default for hints autodiscover. (#36950) · elastic/beats@41ab08c · GitHub](https://github.com/elastic/beats/commit/41ab08cd6aa9e2c367b0c4b670f05f786a2862db#diff-094228d7dfd7e70592349957978802cb8d2162bc76fb96f7379c5b30d61847fd) Which looks like it changed the defaults for autodiscovery to be aimed at Kubernetes. Which leaves me wondering where the code detecting that it should be using Docker config and not Kube is... Since I didn't find any where I was looking. (Not that I know golang beyond just knowing how to program in other languages.)

Anyway, any help would be appreciated.

Thanks!

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [February 16, 2024, 6:47pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/2 "2024-02-16T18:47:51Z")

</div>

Anyone? I'm still stuck on this.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 17, 2024, 11:49pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/3 "2024-02-17T23:49:00Z")

</div>

What happens when you deploy exactly this (do not fix / change to your indenting style)

This is the reference config... problem though will probably be your file extension `*.json-log`

> <https://github.com/elastic/beats/blob/v8.12.0/deploy/docker/filebeat.docker.yml>

When you say it stopped working, what version did you upgrade from?

Can you share the rest of your config .yml? Is there some other provider?

> [@jerrac](#):
>
> I can see filebeat registering the container log files, but it doesn't seem to be pushing the data to Elasticsearch.

There should be some other errors... do you see the connection to Elasticsearch?

You can set logging level to debug as well.

And why are you adding this here?

> [@jerrac](#):
>
> ` host: "unix:///var/run/docker.sock"`

I don't think that is needed/correct if anything, that should be a [processor](https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html)

But is interesting looks like the default [here](https://github.com/elastic/beats/blob/v8.12.0/libbeat/autodiscover/providers/docker/config.go#L48) ( i am not a `go` developer either)

```auto
processors:
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"

```

you could also try the default from the docs

```auto
filebeat.autodiscover.providers:
  - type: docker
    hints.enabled: true
    hints.default_config:
      type: container
      paths:
        - /var/lib/docker/containers/${data.container.id}/*.json-log

```

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [February 20, 2024, 7:48pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/4 "2024-02-20T19:48:01Z")

</div>

I was on 8.11.n prior to the 8.12.0 upgrade that broke autodiscovery.

Filebeat is able to send non-docker container logs to Elasticsearch just fine. So the ES connection is good.

I tried debug logs and didn't spot anything helpful at the time. I'll try again.

I set the autodiscovery host `host: "unix:///var/run/docker.sock"` to the default value just to see if it helped. It didn't.

I have tried the "default\_config" and it didn't work.

I just tried adding the processors config, and it did not help.

Here's my config:

```yaml
processors:
  - add_cloud_metadata: ~
filebeat:
  autodiscover:
    providers:
    - type: docker
      hints.enabled: true
        # - hints.default_config:
        #paths:
        #- /var/lib/docker/containers/${data.container.id}/*.json-log
        #type: container
        #hints.enabled: true
        #host: unix:///var/run/docker.sock
        #type: docker
  config:
    inputs:
      enabled: true
      path: inputs.d/*.yml
      reload.enabled: true
      reload.period: 30s
    modules:
      enabled: true
      path: modules.d/*.yml
      reload.enabled: true
      reload.period: 30s
  modules:
  - audit:
      enabled: false
    auth:
      enabled: true
      var.paths:
      - /host/var/log/auth*
      - /host/var/log/secure*
    module: system
    syslog:
      enabled: true
      var.paths:
      - /host/var/log/messages*
      - /host/var/log/syslog*
output:
  elasticsearch:
    enabled: true
    hosts:
    - https://< user/pass/host >:9200
    index: projectname-logs-%{[agent.version]}
    ssl:
      certificate_authorities:
      - /usr/local/share/ca-certificates/projectname-ca.crt
      enabled: true
      verification_mode: certificate
setup:
  dashboards:
    enabled: false
  ilm:
    enabled: false
  json:
    data_stream: true
    enabled: false
  kibana:
    enabled: false
  template:
    enabled: false
    name: projectname-logs-%{[agent.version]}
    pattern: projectname-logs-%{[agent.version]}-*

```

And journald spit out these logs for filebeat:

```auto
Feb 20 09:59:26 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:26.998-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'log'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.012-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'log'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.012-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.013-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.013-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'error'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.014-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'error'","service.name":"filebeat","ecs.version":"1.6.0"}
Feb 20 09:59:27 swarmnode02 filebeat[3827847]: {"log.level":"warn","@timestamp":"2024-02-20T09:59:27.015-0800","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'access'","service.name":"filebeat","ecs.version":"1.6.0"}

```

I also tried adding:

```yaml
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"

```

to the processors and it did not help. Still get the missing kube field message.

I eventually tried a bunch of things and stripped the config down to:

```yaml
log.level: debug
processors:
  - add_docker_metadata: ~
filebeat:
  autodiscover:
    providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/lib/docker/containers/${data.container.id}/*json.log
output:
  elasticsearch:
    enabled: true
    hosts:
    - https://< user/pass/host >:9200
    index: projectname-logs-%{[agent.version]}
    ssl:
      certificate_authorities:
      - /usr/local/share/ca-certificates/projectname-ca.crt
      enabled: true
      verification_mode: certificate
setup:
  dashboards:
    enabled: false
  ilm:
    enabled: false
  json:
    data_stream: true
    enabled: false
  kibana:
    enabled: false
  template:
    enabled: false
    name: projectname-logs-%{[agent.version]}
    pattern: projectname-logs-%{[agent.version]}-*

```

And I see logs like:

```auto
Feb 20 11:11:36 swarmnode02 filebeat[4053811]: {"log.level":"error","@timestamp":"2024-02-20T11:11:36.713-0800","log.logger":"reader_json","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*JSONReader).decode","file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character '-' after array element","service.name":"filebeat","ecs.version":"1.6.0"}

```

None of the search results I found about json decoding issues helped, and I'm pretty sure it's a red herring.

So, yeah, I'm running in circles at this point.

According to the docs: [Hints based autodiscover | Filebeat Reference [8.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_docker_3) I should be able to just do:

```yaml
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false

```

I just tried that (again, I think...) and did not get anything new.

That, plus the logs about the missing kube field make it clear that Filebeat is trying to configure the access/error logs for some of the containers I have the hint labels on. But instead of looking for docker fields, it's looking for kube fields.

It really feels like a bug to me. Especially since my original config was working prior to the upgrade.

My old config:

```yaml
filebeat:
  autodiscover:
    providers:
    - type: docker
      hints.enabled: true
      host: "unix:///var/run/docker.sock"

```

Any other ideas?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 20, 2024, 7:58pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/5 "2024-02-20T19:58:53Z")

</div>

Nope... Pinged internally...

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [February 20, 2024, 8:43pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/6 "2024-02-20T20:43:27Z")

</div>

Thanks.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 20, 2024, 9:01pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/7 "2024-02-20T21:01:22Z")

</div>

> [@stephenb](#):
>
> `"/var/lib/docker/containers/${data.container.id}/*.json-log"`

Did you try putting that in quotes?

---

<div class="post-metadata">

**Author:** ![GEownt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geownt/32/131957_2.png) [@GEownt](https://discuss.elastic.co/u/GEownt)\
**Post date:** [February 21, 2024, 8:15am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/8 "2024-02-21T08:15:33Z")

</div>

We have nearly exact the same problem, just the error/warnings are different for us.

```auto
autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'

autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths' (source:'/etc/filebeat.yml')

autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'

```

Our config looks like this

```auto
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
            - config:
              - type: container
                paths:
                  - "/var/log/containers/*${data.kubernetes.container.id}.log"
                multiline.pattern: '^[[:space:]]'
                multiline.negate: false
                multiline.match: after

```

We used filebeat 8.11.4 before and everything worked fine, now everything seems to work too but just these messages are spammed into our log system.

---

<div class="post-metadata">

**Author:** ![Michalis\_Katsoulis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michalis_katsoulis/32/93500_2.png) [@Michalis\_Katsoulis](https://discuss.elastic.co/u/Michalis_Katsoulis)\
**Post date:** [February 21, 2024, 8:28am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/9 "2024-02-21T08:28:24Z")

</div>

@jerrac You should use the hints.default\_config like

```auto
hints.default_config:
      type: container
      paths:
        - /var/lib/docker/containers/${data.container.id}/*.log

```

This will override the default config of filestream that was added in 8.12.  
Is the log path you are using `/var/lib/docker/containers/${data.container.id}/*json.log` or `/var/lib/docker/containers/${data.container.id}/*.json-log` because the seconds seems wrong ?

---

<div class="post-metadata">

**Author:** ![Michalis\_Katsoulis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michalis_katsoulis/32/93500_2.png) [@Michalis\_Katsoulis](https://discuss.elastic.co/u/Michalis_Katsoulis)\
**Post date:** [February 21, 2024, 8:44am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/10 "2024-02-21T08:44:06Z")

</div>

@GEownt what are you trying to do with this configuration?  
When hints are used along with templates, then hints will be evaluated only in case there is no template’s condition that resolves to true.  
In your configuration there is a template with no condition, so it will be always true and hints won't be evaluated.  
Are you running filebeat inside a kubernetes cluster?  
Then why not use

```auto
filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
           paths:
           - /var/log/containers/*-${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           - multiline:
              type: pattern
              pattern: '^[[:space:]]'
              negate: false
              match: after
           prospector:
            scanner:
              fingerprint.enabled: true
              symlinks: true
           file_identity.fingerprint: ~

```

Can you try that?

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [February 21, 2024, 10:34pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/11 "2024-02-21T22:34:56Z")

</div>

Log path is `*-json.log`, I had mixed it up previously. Though I still am not getting the logs.

```yaml
filebeat:
  autodiscover:
    providers:
    - hints.default_config:
        paths:
        - /var/lib/docker/containers/${data.container.id}/*.log
        type: container
      hints.enabled: true
      type: docker

```

Seems to have done the trick. Thanks.

I will note that that config will pull in logs from ALL containers. Not just the ones you have labeled with the hints.

Will there be a fix to restore the previous behavior?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 22, 2024, 12:49am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/12 "2024-02-22T00:49:31Z")

</div>

> [@jerrac](#):
>
> `invalid character '-' a`

That was actually buried in there... that is why I asked if you had surrounded with quotes..

Glad you found it

Not yaml professor but when it is a list and there is a `-` in the content... think is should be surrounded in quotes..

---

<div class="post-metadata">

**Author:** ![GEownt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geownt/32/131957_2.png) [@GEownt](https://discuss.elastic.co/u/GEownt)\
**Post date:** [February 22, 2024, 8:16am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/13 "2024-02-22T08:16:40Z")

</div>

I am trying this configuration but then I get plenty of this messages

```auto
{"log.level":"warn","@timestamp":"2024-02-22T08:12:07.984Z","log.logger":"scanner","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles","file.name":"filestream/fswatch.go","file.line":389},"message":"cannot create a file descriptor for an ingest target \"/var/log/containers/calico-node-cz787_calico-system_flexvol-driver-4985acc5f36d15ed5da68fb6795955deef5d06dfbd2a10560135ff8b42597a2d.log\": filesize of \"/var/log/containers/calico-node-cz787_calico-system_flexvol-driver-4985acc5f36d15ed5da68fb6795955deef5d06dfbd2a10560135ff8b42597a2d.log\" is 85 bytes, expected at least 1024 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}

```

and this one still exists

```auto
autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id' (source:'/etc/filebeat.yml')

```

I am using filebeat in my kubernetes cluster.

---

<div class="post-metadata">

**Author:** ![Michalis\_Katsoulis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michalis_katsoulis/32/93500_2.png) [@Michalis\_Katsoulis](https://discuss.elastic.co/u/Michalis_Katsoulis)\
**Post date:** [February 22, 2024, 11:33am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/14 "2024-02-22T11:33:26Z")

</div>

Glad that it is working for you. The previous behaviour won't be restored as the update from container input to filestream was the intention.  
But there should be a fix so that the default\_config does not look for kubernetes variables in the events in case of docker provider.

---

<div class="post-metadata">

**Author:** ![Michalis\_Katsoulis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michalis_katsoulis/32/93500_2.png) [@Michalis\_Katsoulis](https://discuss.elastic.co/u/Michalis_Katsoulis)\
**Post date:** [February 22, 2024, 11:42am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/15 "2024-02-22T11:42:31Z")

</div>

For the first one, it is just a warning that some files are too small in size for filebeat to monitor them. This is due to the [fingerprint mode](https://www.elastic.co/blog/introducing-filestream-fingerprint-mode)

For the second error I will try to reproduce it. Meanwhile can you remove the `data.kubernetes.container.id` part from the `id` in the config block?  
like:  
`id: kubernetes-container-logs-${data.kubernetes.pod.name}`

---

<div class="post-metadata">

**Author:** ![GEownt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geownt/32/131957_2.png) [@GEownt](https://discuss.elastic.co/u/GEownt)\
**Post date:** [February 22, 2024, 1:35pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/16 "2024-02-22T13:35:06Z")

</div>

The error messages are now gone. Is there any way to suspend the fingerprint warning messages? Because they are nearly flooding my logs.

---

<div class="post-metadata">

**Author:** ![Michalis\_Katsoulis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michalis_katsoulis/32/93500_2.png) [@Michalis\_Katsoulis](https://discuss.elastic.co/u/Michalis_Katsoulis)\
**Post date:** [February 22, 2024, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/17 "2024-02-22T14:49:44Z")

</div>

You can either disable the fingerprint feature by removing `fingerprint.enabled` and `file_identity.fingerprint: ~` which is not recommended , either configure it to reduce the fingerprint.length like

```auto
file_identity.fingerprint:
  enabled: false
  offset: 0
  length: 64

```

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [February 22, 2024, 4:06pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/18 "2024-02-22T16:06:44Z")

</div>

> [@Michalis\_Katsoulis](#):
>
> But there should be a fix so that the default\_config does not look for kubernetes variables in the events in case of docker provider.

That's really what I meant, I just worded it badly. 😃

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![GEownt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geownt/32/131957_2.png) [@GEownt](https://discuss.elastic.co/u/GEownt)\
**Post date:** [February 23, 2024, 5:47am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/19 "2024-02-23T05:47:52Z")

</div>

Thanks for your help, now everything works as expected.  
I just get a few warning messages now where the filesize is 0 bytes but thats ok because I can filter them out.

Here is my configuration for reference:

```auto
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}
           paths:
           - /var/log/containers/*${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           - multiline:
              type: pattern
              pattern: '^[[:space:]]'
              negate: false
              match: after
           prospector:
            scanner:
              symlinks: true
              fingerprint:
                enabled: true
                offset: 0
                length: 64
           file_identity.fingerprint: ~

```

---

<div class="post-metadata">

**Author:** ![Michalis\_Katsoulis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michalis_katsoulis/32/93500_2.png) [@Michalis\_Katsoulis](https://discuss.elastic.co/u/Michalis_Katsoulis)\
**Post date:** [February 23, 2024, 9:00am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103/20 "2024-02-23T09:00:20Z")

</div>

I can share some findings with you. The warning message you were receiving `autodiscover: Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'` is a false positive, meaning everything works as expected anyway.  
This was initially a warning message that could happen when unpacking a config while applying the autodiscover config template. You can read it [here](https://github.com/elastic/beats/pull/37816).  
In 8.12.2 this message logging level was brought down to debug so it is not there anymore.  
I would suggest you upgrade to 8.12.2 and set your config like this

```auto
filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
           paths:
           - /var/log/containers/*${data.kubernetes.container.id}.log
           parsers:
           - container: ~
           - multiline:
              type: pattern
              pattern: '^[[:space:]]'
              negate: false
              match: after
           prospector:
            scanner:
              symlinks: true
              fingerprint:
                enabled: true
                offset: 0
                length: 64
           file_identity.fingerprint: ~

```

as you will avoid any chance of data duplication in cases where there are pods with same name. Their container.id though can never be the same, which makes the id field of the config really unique.

[Next page](https://discuss.elastic.co/t/filebeat-docker-autodiscovery-stopped-working-when-i-upgraded-to-8-12-0/353103.md?page=2)