Filebeat: Autodiscover with Nomad + Docker Metadata

[adsr]

Hi everyone,

I am currently struggling with adding the docker metadata to my events which are fetched via the filebeat autodiscover nomad provider.

filebeat.yml

tyfilebeat.autodiscover:
  providers:
    - type: nomad
      node: mynode
      hints.enabled: true
      hints.default_config:
        enabled: true
        type: filestream
        paths:
          - "/opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/secretmessage.*"
        processors:
          - add_fields:
              target: nomad
              fields:
                allocation.id: ${data.nomad.allocation.id}
          - add_fields:
              target: event
              fields:
                dataset: "nomad.log"
                module: "nomad"
..

    processors:
      - add_docker_metadata:
          when:
            equals:
              event.dataset: "nomad.log"
        host: "unix:///var/run/docker.sock"

Activating debug logging yields:

2022-07-12T12:44:32.219+0200    DEBUG   [add_docker_metadata]   add_docker_metadata/add_docker_metadata.go:210  Container not found: cid=alloc

I had a look into beats/add_docker_metadata.go at main · elastic/beats · GitHub and somehow the container id is incorrect (according to the log its just cid but should be mycontainer-some-uuid.

Am I having a configuration issue or is this just not properly determined?
Any ideas are highly appreciated.

[adsr]

After digging further I noticed the following could work in theory:

1. Add a field that represents the container.name to the nomad autodiscover provider:

- add_fields:
    target: "@metadata"
    fields:
      containername: "${data.nomad.task.name}-${data.nomad.allocation.id}"

2. Use this field in the add_docker_metadata processor as match_field, e.g.:

- add_docker_metadata:
    match_fields: ["@metadata.containername"]

This does unfortunately not work as the containername is not the container id and the add_docker_metadata processor is set to only get it via the ID (see beats/add_docker_metadata.go at main · elastic/beats · GitHub l.192).

[adsr]

I found a solution going the other "direction":

1. instruct the nomad docker plugin to write to journald

plugin "docker" {
    config {
        logging {
          type = "journald"
          config { }
        }

2. Collect the container logs from the journal

   type: journald
   id: docker.service
   include_matches:
     - _SYSTEMD_UNIT=docker.service
   processors:
     - drop_event:
         when:
           not:
             has_fields: ['container.id']
     - add_docker_metadata:
          host: "unix:///var/run/docker.sock"
          match_fields: ["container.id"]  # this is the container.id that systemd provided
          match_source: false

3. Enrich the event with the nomad meta data

  - add_nomad_metadata:
      address: http://localhost:4646
      default_indexers.enabled: false
      default_matchers.enabled: false
      node: mynode
      indexers:
        - allocation_uid: # this typo is on purpose, see https://github.com/elastic/beats/issues/29743
       matchers:
        - fields:
            lookup_fields:
              - "container.labels.com_hashicorp_nomad_alloc_id"

et voila we have an event that contains both the docker and nomad metadata.