Autodiscover, docker, and nginx; "Fileset nginx/log is configured but doesn't exist"

[vputz (Vic Putz)]

Okay, almost there! I have autodiscover running correctly and trying to parse logs from my docker containers, including nginx, using this filebeat.yml snippet:

filebeat.config.modules.path: /usr/share/filebeat/modules.d/*.yml

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: "nginx"
          config:
            - module: nginx
              log:
                prospector:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"

...good so far. Except perhaps not. Because when I run filebeat -e -d autodiscover, I see things like

2018-06-20T12:38:08.045Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:109        Got a start event: map[host:172.17.0.2 port:80 docker:{"container":{"id":"ddced2486bf48fc5f2015c1633dce39a94af0da4c56bff1ad8977247454321b7","image":"nginx","labels":{"maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e"},"name":"nginx"}} meta:{"docker":{"container":{"id":"ddced2486bf48fc5f2015c1633dce39a94af0da4c56bff1ad8977247454321b7","image":"nginx","labels":{"maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e"},"name":"nginx"}}} config:[0xc42045b920] start:true], generated configs: [0xc42045b920]
2018-06-20T12:38:08.045Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:193        Got a meta field in the event
2018-06-20T12:38:08.045Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:141        Failed to create runner with config &{{<nil> } <nil> 0xc4200fdfc0}: Fileset nginx/log is configured but doesn't exist

"Fileset nginx/log is configured but doesn't exist" sounds bad. What am I missing here? Frustrating to be this close; it's aaaalmost working!

I should note that "nginx.yml" is copied into /usr/share/filebeat/modules.d, and contains

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

[exekias (Carlos Pérez Aradros)]

Hi @vputz,

You are almost there indeed!

The problem is that log is not a known fileset, it should be either access or error. This is (probably) what you want:

filebeat.config.modules.path: /usr/share/filebeat/modules.d/*.yml

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: "nginx"
          config:
            - module: nginx
              access:
                prospector:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"

Also, FYI: we just released 6.3, which brings hints based autodiscover, have a look to: https://www.elastic.co/blog/docker-and-kubernetes-hints-based-autodiscover-with-beats

Best regards