Kubernetes metadata not showing (add_kubernetes_metadata)

Hi,
Im currently having issues with the add_kubernetes_metadata processor. Im adding a simple config and I've configured a service account to have permissions to query the cluster, but I cant get any metadata entry with k8s info.
My deployment is a pod with filebeat sidecar

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - ${APP_LOG}
  fields_under_root: true
  fields:
    environment: ${CONTEXT_ENVIRONMENT}
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
- type: log
  enabled: true
  paths:
    - ${ACCESS_LOG}
  fields_under_root: true
  fields:
    environment: ${CONTEXT_ENVIRONMENT}
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
output.elasticsearch:
  enabled: true
  hosts:  '${LOGSTASH_URL}'
  index: "test-index-%{+yyyy.MM.dd}"
  compression_level: 3
setup.template.name: "test-index"
setup.template.pattern: "test-index-*"
logging.level: info
logging.selectors: ["*"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_kubernetes_metadata:
      in_cluster: true
      host: ${HOSTNAME}

From the logs I can see that the config is valid and It get some comunications with the cluster

2020-11-16T12:51:55.621Z        INFO    instance/beat.go:611    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-11-16T12:51:55.628Z        INFO    instance/beat.go:618    Beat UUID: 0442cb7c-d3ef-4425-9857-c3d193b39502
2020-11-16T12:51:55.628Z        INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
2020-11-16T12:51:55.628Z        INFO    [beat]  instance/beat.go:931    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "0442cb7c-d3ef-4425-9857-c3d193b39502"}}}
2020-11-16T12:51:55.628Z        INFO    [beat]  instance/beat.go:940    Build info      {"system_info": {"build": {"commit": "6e16f47450373f04d6a60db1d23c5b13b37f7431", "libbeat": "6.8.1", "time": "2019-06-18T12:17:38.000Z", "version": "6.8.1"}}}
2020-11-16T12:51:55.628Z        INFO    [beat]  instance/beat.go:943    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.8"}}}
2020-11-16T12:51:55.630Z        INFO    [beat]  instance/beat.go:947    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-13T11:53:22Z","containerized":true,"name":"log-deployment-b69f89794-lv8m8","ip":["127.0.0.1/8","10.213.196.208/32"],"kernel_version":"3.10.0-1062.12.1.el7.x86_64","mac":["f6:da:62:c4:df:d8"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-11-16T12:51:55.642Z        INFO    [beat]  instance/beat.go:976    Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-11-16T12:51:55.430Z"}}}
2020-11-16T12:51:55.642Z        INFO    instance/beat.go:280    Setup Beat: filebeat; Version: 6.8.1
2020-11-16T12:51:55.651Z        INFO    kubernetes/util.go:71   kubernetes: Using node log-deployment-b69f89794-lv8m8 provided in the config
2020-11-16T12:51:55.654Z        INFO    kubernetes/watcher.go:182       kubernetes: Performing a resource sync for *v1.PodList
2020-11-16T12:51:55.669Z        INFO    kubernetes/watcher.go:198       kubernetes: Resource sync done
2020-11-16T12:51:55.670Z        INFO    elasticsearch/client.go:164     Elasticsearch url: http://10.209.16.12:9200
2020-11-16T12:51:55.670Z        INFO    kubernetes/watcher.go:242       kubernetes: Watching API for resource events
2020-11-16T12:51:55.671Z        INFO    [publisher]     pipeline/module.go:110  Beat name: log-deployment-b69f89794-lv8m8
2020-11-16T12:51:55.674Z        INFO    add_cloud_metadata/add_cloud_metadata.go:345    add_cloud_metadata: hosting provider type detected as ec2, metadata={"availability_zone":"eu-west-1b","instance_id":"i-0abxxxxxxxxxxxxxx","machine_type":"m5.large","provider":"ec2","region":"eu-west-1"}
2020-11-16T12:51:55.674Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2020-11-16T12:51:55.675Z        INFO    instance/beat.go:402    filebeat start running.
2020-11-16T12:51:55.675Z        INFO    registrar/registrar.go:97       No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2020-11-16T12:51:55.679Z        INFO    registrar/registrar.go:134      Loading registrar data from /usr/share/filebeat/data/registry
2020-11-16T12:51:55.679Z        INFO    registrar/registrar.go:141      States Loaded from registrar: 0
2020-11-16T12:51:55.679Z        INFO    crawler/crawler.go:72   Loading Inputs: 2
2020-11-16T12:51:55.680Z        INFO    log/input.go:148        Configured paths: [/var/log/logs-app/app.log]
2020-11-16T12:51:55.680Z        INFO    input/input.go:114      Starting input of type: log; ID: 14367348903407064248
2020-11-16T12:51:55.680Z        INFO    log/input.go:148        Configured paths: [/var/log/logs-app/access.log]
2020-11-16T12:51:55.680Z        INFO    input/input.go:114      Starting input of type: log; ID: 4343579462404243939
2020-11-16T12:51:55.680Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 2
2020-11-16T12:51:55.680Z        INFO    log/harvester.go:255    Harvester started for file: /var/log/logs-app/app.log
2020-11-16T12:51:55.681Z        INFO    log/harvester.go:255    Harvester started for file: /var/log/logs-app/access.log
2020-11-16T12:51:55.681Z        INFO    cfgfile/reload.go:150   Config reloader started
2020-11-16T12:51:55.681Z        INFO    cfgfile/reload.go:205   Loading of config files completed.
2020-11-16T12:51:59.681Z        INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://10.xxx.xx.xx:9200))
2020-11-16T12:51:59.685Z        INFO    elasticsearch/client.go:739     Attempting to connect to Elasticsearch version 6.8.1
2020-11-16T12:51:59.691Z        INFO    template/load.go:128    Template already exists and will not be overwritten.
2020-11-16T12:51:59.691Z        INFO    instance/beat.go:889    Template successfully loaded.
2020-11-16T12:51:59.691Z        INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://10.xxx.xx.xx:9200)) established

With some other config I can see that the cluster is being queried successfuly

2020-11-16T13:30:36.838Z        INFO    kubernetes/util.go:86   kubernetes: Using pod name log-deployment-b69f89794-zllpw and namespace demo-monitoring to discover kubernetes node
2020-11-16T13:30:36.853Z        INFO    kubernetes/util.go:93   kubernetes: Using node ip-10-xxx-xx-xxx.eu-west-1.compute.internal discovered by in cluster pod node query
2020-11-16T13:30:36.853Z        INFO    kubernetes/watcher.go:182       kubernetes: Performing a resource sync for *v1.PodList
2020-11-16T13:30:36.855Z        INFO    kubernetes/watcher.go:198       kubernetes: Resource sync done
2020-11-16T13:30:36.856Z        INFO    kubernetes/watcher.go:242       kubernetes: Watching API for resource events

But still can't see any metadata in the documents

any thoughts or ideas??

Maybe Im using it wrong, but if someone could point me in the right direction It wolud be much appreciated

Did you try enabling debug logs? What kind of Kubernetes cluster is it?

Yes I've enable the log level to debug level in filebeat. The cluster is a EKS

I wonder if the KUBE_CONFIG is accessible. If not, you can set with a property. See docs:

It may happen that logs are in a different location, you can manually change it with logs_path.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.