Filebeat Docker Autodiscovery stopped working when I upgraded to 8.15.1

Elastic Stack Beats
,
1

Hello,

Today I've updated from filebeat 8.11.1 to 8.15.1, but unfortunatly I had to roll back because there was an issue with the autodiscovery option.

From the forums, I've found this post: Filebeat Docker Autodiscovery stopped working when I upgraded to 8.12.0

I'm having the same issue, and after playing with my configuration file I still have issues.

Let's starts at the beginning:

The original configuration is next:

filebeat.config:
  modules:
    path: modules.d/*.yml
    reload.enabled: false

filebeat.modules:
  - module: apache
    access:
      enabled: true
    error:
      enabled: true      
  - module: traefik
    access:
      enabled: true

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false

From the Elastic documentation page, they said this:

image
image869×272 7.29 KB

But unfortunatly, starting 8.12.0 (yeah I tried all major release of filebeat until 8.11.4 to notice it was broken in 8.12.0) it is not working. In fact, the filebeat is running, but it never connects to the logstash server. Using the netstat to see if there are communication between the two, I can confirm that there is nothing.

So based on the other article, I change the configuration for this:

filebeat.autodiscover.providers:
  - type: docker
    hints.enabled: true
    hints.default_config:
      type: container
      paths:
        - /var/lib/docker/containers/${data.container.id}/*-json.log

By changing this, the beat finally connects to the logstash server:

,"message":"Connection to backoff(async(tcp://logstash-prod.elk.domain.com:5554)) established","service.name":"filebeat","ecs.version":"1.6.0"}

I can also see logs from that server being ingested into Elastic.

But now, I'm getting lots of errors into the filebeat's log file:

{"log.level":"error","@timestamp":"2024-09-19T16:10:33.703-0400","log.logger":"reader_json","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*JSONReader).decode","file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character 'i' in literal true (expecting 'r')","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T16:10:48.717-0400","log.logger":"reader_json","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*JSONReader).decode","file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character 'i' in literal true (expecting 'r')","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T16:11:03.730-0400","log.logger":"reader_json","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*JSONReader).decode","file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character 'i' in literal true (expecting 'r')","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T16:11:19.744-0400","log.logger":"reader_json","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*JSONReader).decode","file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character 'i' in literal true (expecting 'r')","service.name":"filebeat","ecs.version":"1.6.0"}

Once again, in version 8.11.1, using the standard configuration, I do not have those problems.

I'm wondering if that workaround I've done is the only way to make it working, or if there is something else I can do to get rid of those errors from the filebeat's log file.

P.S. I opened up a case at Elastic, will see what they will tell me.

Thanks!

Yanick