Autodiscover with nomad doesn't work

Natanael_Weslley · November 10, 2022, 7:52pm

Hi, guys!

I'm having some trouble to configure autodiscover to run with nomad, to get allocation logs from the nomad agent and send to logstash. Everytime i got this message in filebeat logs.
// "message":"Non-zero metrics in the last 30s"
but in nomad a have an job that is generating logs every 60 seconds.

This is my filebeat config:

#### debug mode
loggin.info: debug

#### logstash output
output.logstash:
  hosts: [ "xxx.xxx.xxx.xxx:5055"]
  slow_start: true

### autodicover
filebeat.autodiscover:
  providers:
    - type: nomad
      scope: cluster
      hints.enabled: true
      allow_stale: true
      templates:
        config:
          - type: log
            paths:
              - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/*

This is the filebeat logs:

{"log.level":"info","@timestamp":"2022-11-10T19:35:15.049Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.5.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.054Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ip-10-202-9-107.ec2.internal","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.055Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-10T19:35:15.056Z","log.origin":{"file.name":"beater/filebeat.go","file.line":162},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.057Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.058Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.059Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-10T19:35:15.059Z","log.origin":{"file.name":"beater/filebeat.go","file.line":288},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.060Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.061Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.061Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-10T19:35:15.062Z","log.logger":"cfgwarn","log.origin":{"file.name":"nomad/nomad.go","file.line":58},"message":"EXPERIMENTAL: The nomad autodiscover provider is experimental.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.063Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.064Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:15.696Z","log.logger":"nomad","log.origin":{"file.name":"nomad/watcher.go","file.line":192},"message":"Watching API for resource events","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T19:35:45.066Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"filebeat.service"},"cpuacct":{"id":"filebeat.service","total":{"ns":650388214}},"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":33452032}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":640,"time":{"ms":640},"value":640},"user":{"ticks":570,"time":{"ms":570}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":10},"info":{"ephemeral_id":"6aa6ed52-c270-4b8e-8f35-48feb7d77ebd","name":"filebeat","uptime":{"ms":30155},"version":"8.5.0"},"memstats":{"gc_next":21512168,"memory_alloc":15031152,"memory_sys":28918792,"memory_total":69080704,"rss":114196480},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.09,"15":0.67,"5":0.38,"norm":{"1":0.045,"15":0.335,"5":0.19}}}},"ecs.version":"1.6.0"}}

This is my nomad job:

job "template" {
    datacenters = ["us-east-1"]
    type = "batch"
    periodic {
        cron  = "* * * * *"
    }
    group "group" {
        count = 1
        task "command" {
            driver = "exec"
            config {
                command = "bash"
                args = ["-c", "cat local/template.out"]
            }
            template {
                data = <<EOH
                        node.unique.id: {{ env "node.unique.id" }}
                        node.datacenter: {{ env "node.datacenter" }}
                        node.unique.name: {{ env "node.unique.name" }}
                        node.class: {{ env "node.class" }}
                        attr.cpu.arch: {{ env "attr.cpu.arch" }}
                        attr.cpu.numcores: {{ env "attr.cpu.numcores" }}
                        attr.cpu.totalcompute: {{ env "attr.cpu.totalcompute" }}
                        attr.consul.datacenter: {{ env "attr.consul.datacenter" }}
                        attr.unique.hostname: {{ env "attr.unique.hostname" }}
                        attr.unique.network.ip-address: {{ env "attr.unique.network.ip-address" }}
                        attr.kernel.name: {{ env "attr.kernel.name" }}
                        attr.kernel.version: {{ env "attr.kernel.version" }}
                        attr.platform.aws.ami-id: {{ env "attr.platform.aws.ami-id" }}
                        attr.platform.aws.instance-type: {{ env "attr.platform.aws.instance-type" }}
                        attr.os.name: {{ env "attr.os.name" }}
                        attr.os.version: {{ env "attr.os.version" }}
                        NOMAD_ALLOC_DIR: {{env "NOMAD_ALLOC_DIR"}}
                        NOMAD_TASK_DIR: {{env "NOMAD_TASK_DIR"}}
                        NOMAD_SECRETS_DIR: {{env "NOMAD_SECRETS_DIR"}}
                        NOMAD_MEMORY_LIMIT: {{env "NOMAD_MEMORY_LIMIT"}}
                        NOMAD_CPU_LIMIT: {{env "NOMAD_CPU_LIMIT"}}
                        NOMAD_ALLOC_ID: {{env "NOMAD_ALLOC_ID"}}
                        NOMAD_ALLOC_NAME: {{env "NOMAD_ALLOC_NAME"}}
                        NOMAD_ALLOC_INDEX: {{env "NOMAD_ALLOC_INDEX"}}
                        NOMAD_TASK_NAME: {{env "NOMAD_TASK_NAME"}}
                        NOMAD_GROUP_NAME: {{env "NOMAD_GROUP_NAME"}}
                        NOMAD_JOB_NAME: {{env "NOMAD_JOB_NAME"}}
                        NOMAD_DC: {{env "NOMAD_DC"}}
                        NOMAD_REGION: {{env "NOMAD_REGION"}}
                        VAULT_TOKEN: {{env "VAULT_TOKEN"}}
                        GOMAXPROCS: {{env "GOMAXPROCS"}}
                        HOME: {{env "HOME"}}
                        LANG: {{env "LANG"}}
                        LOGNAME: {{env "LOGNAME"}}
                        NOMAD_ADDR_export: {{env "NOMAD_ADDR_export"}}
                        NOMAD_ADDR_exstat: {{env "NOMAD_ADDR_exstat"}}
                        NOMAD_ALLOC_DIR: {{env "NOMAD_ALLOC_DIR"}}
                        NOMAD_ALLOC_ID: {{env "NOMAD_ALLOC_ID"}}
                        NOMAD_ALLOC_INDEX: {{env "NOMAD_ALLOC_INDEX"}}
                        NOMAD_ALLOC_NAME: {{env "NOMAD_ALLOC_NAME"}}
                        NOMAD_CPU_LIMIT: {{env "NOMAD_CPU_LIMIT"}}
                        NOMAD_DC: {{env "NOMAD_DC"}}
                        NOMAD_GROUP_NAME: {{env "NOMAD_GROUP_NAME"}}
                        NOMAD_HOST_PORT_export: {{env "NOMAD_HOST_PORT_export"}}
                        NOMAD_HOST_PORT_exstat: {{env "NOMAD_HOST_PORT_exstat"}}
                        NOMAD_IP_export: {{env "NOMAD_IP_export"}}
                        NOMAD_IP_exstat: {{env "NOMAD_IP_exstat"}}
                        NOMAD_JOB_NAME: {{env "NOMAD_JOB_NAME"}}
                        NOMAD_MEMORY_LIMIT: {{env "NOMAD_MEMORY_LIMIT"}}
                        NOMAD_PORT_export: {{env "NOMAD_PORT_export"}}
                        NOMAD_PORT_exstat: {{env "NOMAD_PORT_exstat"}}
                        NOMAD_REGION: {{env "NOMAD_REGION"}}
                        NOMAD_SECRETS_DIR: {{env "NOMAD_SECRETS_DIR"}}
                        NOMAD_TASK_DIR: {{env "NOMAD_TASK_DIR"}}
                        NOMAD_TASK_NAME: {{env "NOMAD_TASK_NAME"}}
                        PATH: {{env "PATH"}}
                        PWD: {{env "PWD"}}
                        SHELL: {{env "SHELL"}}
                        SHLVL: {{env "SHLVL"}}
                        USER: {{env "USER"}}
                        VAULT_TOKEN: {{env "VAULT_TOKEN"}}
                        concat key:  service/fabio/{{ env "NOMAD_JOB_NAME" }}/listeners
                        key: {{ keyOrDefault ( printf "service/fabio/%s/listeners" ( env "NOMAD_JOB_NAME" ) ) ":9999" }} {{ define "custom" }}service/fabio/{{env "NOMAD_JOB_NAME" }}/listeners{{ end }}
                        key: {{ keyOrDefault (executeTemplate "custom") ":9999" }} math - alloc_id + 1: {{env "NOMAD_ALLOC_INDEX" | parseInt | add 1}}
                        EOH
                destination = "local/template.out"
            }
        }
    }
}

Can you help me understand why autodiscover doesn't work with nomad?

andrewkroh · November 10, 2022, 11:59pm

I'm not sure if this helps, but I run a Filebeat container as a system type job in Nomad and this is my autodiscover config:

filebeat.autodiscover:
  providers:
    - type: nomad
      address: https://{{ env "attr.unique.network.ip-address" }}:4646
      node: '{{ env "node.unique.name" }}'
      secret_id: '${NOMAD_SECRET_ID:}'
      hints.enabled: true
      hints.default_config:
        # Require label co.elastic.logs/enabled=true.
        enabled: false
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
        processors:
          - add_fields:
              target: nomad
              fields:
                allocation:
                  id: ${data.nomad.allocation.id}
          - if:
              contains.log.file.path: '.stderr'
            then:
              - add_fields:
                  target: log
                  fields:
                    stream: stderr
            else:
              - add_fields:
                  target: log
                  fields:
                    stream: stdout

These env vars are injected into the container.

      env {
        # This is used by the Go Nomad API client in Filebeat.
        NOMAD_CACERT = "${NOMAD_SECRETS_DIR}/certs/my-ca.crt"

        # Filebeat connects to the fingerprinted IP which may not
        # be valid for the cert.
        # https://discuss.hashicorp.com/t/how-to-expose-nomad-http-api-within-a-docker-task-without-using-host-networking/34615
        NOMAD_SKIP_VERIFY = "true"
      }

Jobs that I want monitored must opt into log collection with:

      meta {
        "co.elastic.logs/enabled" = true
      }

Natanael_Weslley · November 17, 2022, 8:58pm

Hi, andrewkroh!

I'm trying to catch logs from nomad agents but autodiscover doesn't out put the logs.

I install filebeat on the nomad agent and then a configure a manual input to get logs from agent, like this:

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /opt/nomad/alloc/a5172f8a-1ecb-e5c6-6530-8cc512b628c4/alloc/logs/*

however, when i try to use the autodiscover the logs are not sent. I adapted your config to me, but i had no success.

filebeat.autodiscover:
  providers:
    - type: nomad
      address: http://127.0.0.1:4646
      node: mynode
      hints.enabled: true
      hints.default_config:
        enabled: false
        type: log
        paths:
          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
        processors:
          - add_fields:
              target: nomad
              fields:
                allocation:
                  id: ${data.nomad.allocation.id}
          - if:
              contains.log.file.path: '.stderr'
            then:
              - add_fields:
                  target: log
                  fields:
                    stream: stderr
            else:
              - add_fields:
                  target: log
                  fields:
                    stream: stdout

output.file:
  path: "/tmp"
  filename: output.log

i have a doubt! I was looking for possibilities in github, i found this: Add support for Nomad · Issue #16853 · elastic/beats · GitHub

And i see this in logs: "message":"EXPERIMENTAL: The nomad autodiscover provider is experimental."

Is the autodiscover unable to catch logs from nomad?

andrewkroh · November 17, 2022, 11:30pm

Are you trying to get the logs from the agent itself? Or are you trying to read logs produced by nomad jobs?

If it's the jobs, then have you added a label to those jobs? You would need to add this to each job:

      meta {
        "co.elastic.logs/enabled" = true
      }

Natanael_Weslley · November 18, 2022, 5:40pm

l'm trying to get logs produced by nomad jobs.

I got this error when i try to use meta:

$ nomad run template.hcl 
Error getting job struct: Failed to parse using HCL 2. Use the HCL 1 parser with `nomad run -hcl1`, or address the following issues:
template.hcl:8,9-10: Invalid argument name; Argument names must not be quoted.
$ nomad run -hcl1 template.hcl 
Invalid job options: cannot parse job file as HCLv1 and HCLv2 strict.

how to solve? Or can you give an example on how to use meta? the documentation is not clearly explained

andrewkroh · November 18, 2022, 8:11pm

I never encountered that issue. I store my jobs as hcl v1 and then use terraform to submit them. My guess is that terraform converts the hcl to json before submitting them and it might be more lenient in parsing the hcl.

// nomad.tf
resource "nomad_job" "traefik" {
  jobspec = file("${path.module}/traefik.hcl")
}

system · December 16, 2022, 10:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.