Filebeat Autodiscover appenders - configuration not applied

Elastic Stack Beats
,
1

Hi everyone, before opening an issue on Github, I will try here. As subject states, Filebeat Autodiscover appenders are not working as expected - configuration is not being applied. If I configure add_docker_metadata as label on service, Docker Labels are being applied, using autodiscover plugin config hints.

I am running ELK stack on Mac, using Docker Desktop for Mac.
Filebeat version is 7.9.1.
Anyhow, here goes my minimal filebeat.yml config:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  
  
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths: 
          - '/var/lib/docker/containers/${data.container.id}/*.log'
        combine_partial: false
        fields_under_root: true
        exclude_files: [".gz$", "_[0-9]{4}-[0-9]{2}-[0-9]{2}"]
        tail_files: false
        ignore_older: 24h
        close_inactive: 1h
      appenders:
        - type: config
          merge: append
          config:
            - add_docker_metadata:
                labels.dedot: false

output.elasticsearch:
  hosts: 'http://es01:9200'
  pipeline: "filebeat-7.9.1-pipeline"

For testing purpose, I am using a docker-compose to spin up another service:

version: '3.2'
services: 
  docker-logs: 
    user: root
    image: docker-logs:latest
    container_name: docker-logs
    labels:
      # co.elastic.logs/processors.add_docker_metadata.labels.dedot: "false"
      co.elastic.logs/processors.add_fields.fields.name: "MY_FIELDS"
      co.elastic.logs/processors.add_fields.fields.id: "MY_VAULE"

However, the Docker labels are only being applied, if I configure the add_docker_metadata on service above, where desired would be to apply the add_docker_metadata processor in autodiscover context of filebeat.yml configuration file.

From Filebeat logs there is only one line being logged, relevant to appenders:

filebeat    | 2020-10-19T17:18:53.562Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
filebeat    | 2020-10-19T17:18:53.577Z	WARN	[cfgwarn]	config/config.go:50	BETA: The config appender is beta
filebeat    | 2020-10-19T17:18:53.583Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager

EDITS: Mostly code formatting.

2

If I understand your use case correctly, I think you can just use "regular" autodiscover and don't actually need hints-based autodiscover.

3

I would disagree. At least from the documentation perspective. What if I want to add some fields using add_fields in context of autodiscover?

4

Ah, I see, you want the additional fields to vary dynamically per service. Understood now. In that case, yes, you'll want hints-based autodiscover.

Let me try to reproduce your situation locally and get back to you.

1 Like
5

Ideally yes, but also it should be applicable in case that condition isn't being specified.

Let me know if you need more details, but it should be pretty straightforward to reproduce the issue.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.