Can filebeat.autodiscover load external configuration files, like inputs and modules can?

[DougR]

Title says it all.

Filebeat has the capability to load external configuration files for inputs and modules, which is documented here.

Does this capability also exist for autodiscover configurations? I suspect the answer is "no," but want to confirm.

I tested it with the Elastic-provided Filebeat helm chart and the following configuration:

---
filebeat:
  autodiscover:
    providers:
      path: '${path.config}/provider-*.yml'
      reload:
        enabled: false
output:
  ...

and ensured that the provider files were in the correct place in the container. Filebeat started normally, and without errors, but did not load the provider files.

[jsoriano (Jaime Soriano)]

Hey @DougR,

You are right, this is not possible. We recently discussed about this here https://github.com/elastic/beats/issues/22067, and in principle we are not going to implement this.

What would be your use case?

[DougR]

Unless I misread it, I'm not quite certain these are discussing the same thing.

The primary use case is that it simplifies maintenance. I use the Elastic-provided helm chart to deploy my filebeat as a daemonset to my Kubernetes cluster. We have numerous applications running on the same cluster, in separate namespaces. Currently, it is necessary to maintain a single filebeat.yml with all providers. This will allow me to maintain them as individual files, making them more manageable, similar to modules, and inputs. I.e., in my chart values for my filebeat, I would be able to include the following

---
filebeatConfig:
  filebeat.yml: |
    ---
    filebeat:
      autodiscover:
        providers:
          path: '${path.config}/provider-*.yml'
          reload:
            enabled: false
    output:
      ...

  provider-001.yml: |
    ...

  provider-002.yml: |
    ...

  provider-003.yml: |
    ...

and ensured that the provider files were in the correct place in the container. Filebeat started normally, and without errors, but did not load the provider files.

When using Ansible to manage the deployment, it allows me to load the provider files into an ansible dict object and do the following with a filebeat.yml.j2 template:

filebeatConfig:
  filebeat.yml: |
    ---
    filebeat:
    autodiscover:
      providers:
        path: '${path.config}/provider-*.yml'
        reload:
          enabled: false
    output:
      ...

{% for item in filebeat_providers.keys()|default([])|sort %}
  {{ item }}: |
    {{ filebeat_providers[item]|indent(width=4) }}

{% endfor %}

While it is possible to insert them into the filebeat.yml in a similar fashion, which is what I currently do, allowing them to be read in as separate files is more fool-proof and easier to troubleshoot.

While I haven't looked at it before now, I think that hint-based configuration may be a good future solution for us. However, we are still early enough in adopting our strategy that I don't know that it's feasible at the moment.