# Autodiscover with nomad doesn't work

**URL:** https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690
**Category:** Beats
**Tags:** filebeat
**Created:** [November 10, 2022, 7:52pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690 "2022-11-10T19:52:47Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Natanael\_Weslley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/natanael_weslley/32/113240_2.png) [@Natanael\_Weslley](https://discuss.elastic.co/u/Natanael_Weslley)
#### Post date: [November 10, 2022, 7:52pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/1 "2022-11-10T19:52:47Z")

</div>

Hi, guys!

I'm having some trouble to configure autodiscover to run with nomad, to get allocation logs from the nomad agent and send to logstash. Everytime i got this message in filebeat logs.  
// "message":"Non-zero metrics in the last 30s"  
but in nomad a have an job that is generating logs every 60 seconds.

This is my filebeat config:

```auto
#### debug mode
loggin.info: debug

#### logstash output
output.logstash:
  hosts: ["xxx.xxx.xxx.xxx:5055"]
  slow_start: true

### autodicover
filebeat.autodiscover:
  providers:
    - type: nomad
      scope: cluster
      hints.enabled: true
      allow_stale: true
      templates:
        config:
          - type: log
            paths:
              - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/*

```

This is the filebeat logs:

```auto
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.049Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.5.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.054Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ip-10-202-9-107.ec2.internal","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.055Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-10T19:35:15.056Z","log.origin":{"file.name":"beater/filebeat.go","file.line":162},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.057Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.058Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.059Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-10T19:35:15.059Z","log.origin":{"file.name":"beater/filebeat.go","file.line":288},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.060Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.061Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.061Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-10T19:35:15.062Z","log.logger":"cfgwarn","log.origin":{"file.name":"nomad/nomad.go","file.line":58},"message":"EXPERIMENTAL: The nomad autodiscover provider is experimental.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.063Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.064Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.696Z","log.logger":"nomad","log.origin":{"file.name":"nomad/watcher.go","file.line":192},"message":"Watching API for resource events","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:45.066Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"filebeat.service"},"cpuacct":{"id":"filebeat.service","total":{"ns":650388214}},"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":33452032}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":640,"time":{"ms":640},"value":640},"user":{"ticks":570,"time":{"ms":570}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":10},"info":{"ephemeral_id":"6aa6ed52-c270-4b8e-8f35-48feb7d77ebd","name":"filebeat","uptime":{"ms":30155},"version":"8.5.0"},"memstats":{"gc_next":21512168,"memory_alloc":15031152,"memory_sys":28918792,"memory_total":69080704,"rss":114196480},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.09,"15":0.67,"5":0.38,"norm":{"1":0.045,"15":0.335,"5":0.19}}}},"ecs.version":"1.6.0"}}

```

This is my nomad job:

```auto
job "template" {
    datacenters = ["us-east-1"]
    type = "batch"
    periodic {
        cron = "* * * * *"
    }
    group "group" {
        count = 1
        task "command" {
            driver = "exec"
            config {
                command = "bash"
                args = ["-c", "cat local/template.out"]
            }
            template {
                data = <<EOH
                        node.unique.id: {{ env "node.unique.id" }}
                        node.datacenter: {{ env "node.datacenter" }}
                        node.unique.name: {{ env "node.unique.name" }}
                        node.class: {{ env "node.class" }}
                        attr.cpu.arch: {{ env "attr.cpu.arch" }}
                        attr.cpu.numcores: {{ env "attr.cpu.numcores" }}
                        attr.cpu.totalcompute: {{ env "attr.cpu.totalcompute" }}
                        attr.consul.datacenter: {{ env "attr.consul.datacenter" }}
                        attr.unique.hostname: {{ env "attr.unique.hostname" }}
                        attr.unique.network.ip-address: {{ env "attr.unique.network.ip-address" }}
                        attr.kernel.name: {{ env "attr.kernel.name" }}
                        attr.kernel.version: {{ env "attr.kernel.version" }}
                        attr.platform.aws.ami-id: {{ env "attr.platform.aws.ami-id" }}
                        attr.platform.aws.instance-type: {{ env "attr.platform.aws.instance-type" }}
                        attr.os.name: {{ env "attr.os.name" }}
                        attr.os.version: {{ env "attr.os.version" }}
                        NOMAD_ALLOC_DIR: {{env "NOMAD_ALLOC_DIR"}}
                        NOMAD_TASK_DIR: {{env "NOMAD_TASK_DIR"}}
                        NOMAD_SECRETS_DIR: {{env "NOMAD_SECRETS_DIR"}}
                        NOMAD_MEMORY_LIMIT: {{env "NOMAD_MEMORY_LIMIT"}}
                        NOMAD_CPU_LIMIT: {{env "NOMAD_CPU_LIMIT"}}
                        NOMAD_ALLOC_ID: {{env "NOMAD_ALLOC_ID"}}
                        NOMAD_ALLOC_NAME: {{env "NOMAD_ALLOC_NAME"}}
                        NOMAD_ALLOC_INDEX: {{env "NOMAD_ALLOC_INDEX"}}
                        NOMAD_TASK_NAME: {{env "NOMAD_TASK_NAME"}}
                        NOMAD_GROUP_NAME: {{env "NOMAD_GROUP_NAME"}}
                        NOMAD_JOB_NAME: {{env "NOMAD_JOB_NAME"}}
                        NOMAD_DC: {{env "NOMAD_DC"}}
                        NOMAD_REGION: {{env "NOMAD_REGION"}}
                        VAULT_TOKEN: {{env "VAULT_TOKEN"}}
                        GOMAXPROCS: {{env "GOMAXPROCS"}}
                        HOME: {{env "HOME"}}
                        LANG: {{env "LANG"}}
                        LOGNAME: {{env "LOGNAME"}}
                        NOMAD_ADDR_export: {{env "NOMAD_ADDR_export"}}
                        NOMAD_ADDR_exstat: {{env "NOMAD_ADDR_exstat"}}
                        NOMAD_ALLOC_DIR: {{env "NOMAD_ALLOC_DIR"}}
                        NOMAD_ALLOC_ID: {{env "NOMAD_ALLOC_ID"}}
                        NOMAD_ALLOC_INDEX: {{env "NOMAD_ALLOC_INDEX"}}
                        NOMAD_ALLOC_NAME: {{env "NOMAD_ALLOC_NAME"}}
                        NOMAD_CPU_LIMIT: {{env "NOMAD_CPU_LIMIT"}}
                        NOMAD_DC: {{env "NOMAD_DC"}}
                        NOMAD_GROUP_NAME: {{env "NOMAD_GROUP_NAME"}}
                        NOMAD_HOST_PORT_export: {{env "NOMAD_HOST_PORT_export"}}
                        NOMAD_HOST_PORT_exstat: {{env "NOMAD_HOST_PORT_exstat"}}
                        NOMAD_IP_export: {{env "NOMAD_IP_export"}}
                        NOMAD_IP_exstat: {{env "NOMAD_IP_exstat"}}
                        NOMAD_JOB_NAME: {{env "NOMAD_JOB_NAME"}}
                        NOMAD_MEMORY_LIMIT: {{env "NOMAD_MEMORY_LIMIT"}}
                        NOMAD_PORT_export: {{env "NOMAD_PORT_export"}}
                        NOMAD_PORT_exstat: {{env "NOMAD_PORT_exstat"}}
                        NOMAD_REGION: {{env "NOMAD_REGION"}}
                        NOMAD_SECRETS_DIR: {{env "NOMAD_SECRETS_DIR"}}
                        NOMAD_TASK_DIR: {{env "NOMAD_TASK_DIR"}}
                        NOMAD_TASK_NAME: {{env "NOMAD_TASK_NAME"}}
                        PATH: {{env "PATH"}}
                        PWD: {{env "PWD"}}
                        SHELL: {{env "SHELL"}}
                        SHLVL: {{env "SHLVL"}}
                        USER: {{env "USER"}}
                        VAULT_TOKEN: {{env "VAULT_TOKEN"}}
                        concat key: service/fabio/{{ env "NOMAD_JOB_NAME" }}/listeners
                        key: {{ keyOrDefault ( printf "service/fabio/%s/listeners" ( env "NOMAD_JOB_NAME" ) ) ":9999" }} {{ define "custom" }}service/fabio/{{env "NOMAD_JOB_NAME" }}/listeners{{ end }}
                        key: {{ keyOrDefault (executeTemplate "custom") ":9999" }} math - alloc_id + 1: {{env "NOMAD_ALLOC_INDEX" | parseInt | add 1}}
                        EOH
                destination = "local/template.out"
            }
        }
    }
}

```

Can you help me understand why autodiscover doesn't work with nomad?

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 10, 2022, 11:59pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/2 "2022-11-10T23:59:05Z")

</div>

I'm not sure if this helps, but I run a Filebeat container as a system type job in Nomad and this is my autodiscover config:

```auto
filebeat.autodiscover:
  providers:
    - type: nomad
      address: https://{{ env "attr.unique.network.ip-address" }}:4646
      node: '{{ env "node.unique.name" }}'
      secret_id: '${NOMAD_SECRET_ID:}'
      hints.enabled: true
      hints.default_config:
        # Require label co.elastic.logs/enabled=true.
        enabled: false
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
        processors:
          - add_fields:
              target: nomad
              fields:
                allocation:
                  id: ${data.nomad.allocation.id}
          - if:
              contains.log.file.path: '.stderr'
            then:
              - add_fields:
                  target: log
                  fields:
                    stream: stderr
            else:
              - add_fields:
                  target: log
                  fields:
                    stream: stdout

```

These env vars are injected into the container.

```auto
      env {
        # This is used by the Go Nomad API client in Filebeat.
        NOMAD_CACERT = "${NOMAD_SECRETS_DIR}/certs/my-ca.crt"

        # Filebeat connects to the fingerprinted IP which may not
        # be valid for the cert.
        # https://discuss.hashicorp.com/t/how-to-expose-nomad-http-api-within-a-docker-task-without-using-host-networking/34615
        NOMAD_SKIP_VERIFY = "true"
      }

```

Jobs that I want monitored must opt into log collection with:

```auto
      meta {
        "co.elastic.logs/enabled" = true
      }

```

---

<div class="post-metadata">

### Author: ![Natanael\_Weslley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/natanael_weslley/32/113240_2.png) [@Natanael\_Weslley](https://discuss.elastic.co/u/Natanael_Weslley)
#### Post date: [November 17, 2022, 8:58pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/3 "2022-11-17T20:58:28Z")

</div>

Hi, andrewkroh!

I'm trying to catch logs from nomad agents but autodiscover doesn't out put the logs.

I install filebeat on the nomad agent and then a configure a manual input to get logs from agent, like this:

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /opt/nomad/alloc/a5172f8a-1ecb-e5c6-6530-8cc512b628c4/alloc/logs/*

```

however, when i try to use the autodiscover the logs are not sent. I adapted your config to me, but i had no success.

```auto
filebeat.autodiscover:
  providers:
    - type: nomad
      address: http://127.0.0.1:4646
      node: mynode
      hints.enabled: true
      hints.default_config:
        enabled: false
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
        processors:
          - add_fields:
              target: nomad
              fields:
                allocation:
                  id: ${data.nomad.allocation.id}
          - if:
              contains.log.file.path: '.stderr'
            then:
              - add_fields:
                  target: log
                  fields:
                    stream: stderr
            else:
              - add_fields:
                  target: log
                  fields:
                    stream: stdout

output.file:
  path: "/tmp"
  filename: output.log

```

i have a doubt! I was looking for possibilities in github, i found this: [Add support for Nomad · Issue #16853 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/16853)

And i see this in logs: `"message":"EXPERIMENTAL: The nomad autodiscover provider is experimental."`

Is the autodiscover unable to catch logs from nomad?

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 17, 2022, 11:30pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/4 "2022-11-17T23:30:23Z")

</div>

> [@Natanael\_Weslley](#):
>
> I'm trying to catch logs from nomad agents

Are you trying to get the logs from the agent itself? Or are you trying to read logs produced by nomad jobs?

If it's the jobs, then have you added a label to those jobs? You would need to add this to each job:

```auto
      meta {
        "co.elastic.logs/enabled" = true
      }

```

---

<div class="post-metadata">

### Author: ![Natanael\_Weslley](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/natanael_weslley/32/113240_2.png) [@Natanael\_Weslley](https://discuss.elastic.co/u/Natanael_Weslley)
#### Post date: [November 18, 2022, 5:40pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/5 "2022-11-18T17:40:59Z")

</div>

> [@andrewkroh](#):
>
> Are you trying to get the logs from the agent itself? Or are you trying to read logs produced by nomad jobs?

l'm trying to get logs produced by nomad jobs.

I got this error when i try to use meta:

```auto
$ nomad run template.hcl 
Error getting job struct: Failed to parse using HCL 2. Use the HCL 1 parser with `nomad run -hcl1`, or address the following issues:
template.hcl:8,9-10: Invalid argument name; Argument names must not be quoted.
$ nomad run -hcl1 template.hcl 
Invalid job options: cannot parse job file as HCLv1 and HCLv2 strict.

```

how to solve? Or can you give an example on how to use meta? the documentation is not clearly explained 😅

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 18, 2022, 8:11pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/6 "2022-11-18T20:11:16Z")

</div>

I never encountered that issue. I store my jobs as hcl v1 and then use terraform to submit them. My guess is that terraform converts the hcl to json before submitting them and it might be more lenient in parsing the hcl.

```auto
// nomad.tf
resource "nomad_job" "traefik" {
  jobspec = file("${path.module}/traefik.hcl")
}

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 16, 2022, 10:11pm UTC](https://discuss.elastic.co/t/autodiscover-with-nomad-doesnt-work/318690/7 "2022-12-16T22:11:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.