Hi,
Im currently having issues with the add_kubernetes_metadata processor. Im adding a simple config and I've configured a service account to have permissions to query the cluster, but I cant get any metadata entry with k8s info.
My deployment is a pod with filebeat sidecar
filebeat.inputs:
- type: log
enabled: true
paths:
- ${APP_LOG}
fields_under_root: true
fields:
environment: ${CONTEXT_ENVIRONMENT}
json.keys_under_root: true
json.overwrite_keys: false
json.add_error_key: true
- type: log
enabled: true
paths:
- ${ACCESS_LOG}
fields_under_root: true
fields:
environment: ${CONTEXT_ENVIRONMENT}
json.keys_under_root: true
json.overwrite_keys: false
json.add_error_key: true
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.elasticsearch:
enabled: true
hosts: '${LOGSTASH_URL}'
index: "test-index-%{+yyyy.MM.dd}"
compression_level: 3
setup.template.name: "test-index"
setup.template.pattern: "test-index-*"
logging.level: info
logging.selectors: ["*"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_kubernetes_metadata:
in_cluster: true
host: ${HOSTNAME}
From the logs I can see that the config is valid and It get some comunications with the cluster
2020-11-16T12:51:55.621Z INFO instance/beat.go:611 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-11-16T12:51:55.628Z INFO instance/beat.go:618 Beat UUID: 0442cb7c-d3ef-4425-9857-c3d193b39502
2020-11-16T12:51:55.628Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2020-11-16T12:51:55.628Z INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "0442cb7c-d3ef-4425-9857-c3d193b39502"}}}
2020-11-16T12:51:55.628Z INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "6e16f47450373f04d6a60db1d23c5b13b37f7431", "libbeat": "6.8.1", "time": "2019-06-18T12:17:38.000Z", "version": "6.8.1"}}}
2020-11-16T12:51:55.628Z INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.8"}}}
2020-11-16T12:51:55.630Z INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-13T11:53:22Z","containerized":true,"name":"log-deployment-b69f89794-lv8m8","ip":["127.0.0.1/8","10.213.196.208/32"],"kernel_version":"3.10.0-1062.12.1.el7.x86_64","mac":["f6:da:62:c4:df:d8"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-11-16T12:51:55.642Z INFO [beat] instance/beat.go:976 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-11-16T12:51:55.430Z"}}}
2020-11-16T12:51:55.642Z INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.8.1
2020-11-16T12:51:55.651Z INFO kubernetes/util.go:71 kubernetes: Using node log-deployment-b69f89794-lv8m8 provided in the config
2020-11-16T12:51:55.654Z INFO kubernetes/watcher.go:182 kubernetes: Performing a resource sync for *v1.PodList
2020-11-16T12:51:55.669Z INFO kubernetes/watcher.go:198 kubernetes: Resource sync done
2020-11-16T12:51:55.670Z INFO elasticsearch/client.go:164 Elasticsearch url: http://10.209.16.12:9200
2020-11-16T12:51:55.670Z INFO kubernetes/watcher.go:242 kubernetes: Watching API for resource events
2020-11-16T12:51:55.671Z INFO [publisher] pipeline/module.go:110 Beat name: log-deployment-b69f89794-lv8m8
2020-11-16T12:51:55.674Z INFO add_cloud_metadata/add_cloud_metadata.go:345 add_cloud_metadata: hosting provider type detected as ec2, metadata={"availability_zone":"eu-west-1b","instance_id":"i-0abxxxxxxxxxxxxxx","machine_type":"m5.large","provider":"ec2","region":"eu-west-1"}
2020-11-16T12:51:55.674Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2020-11-16T12:51:55.675Z INFO instance/beat.go:402 filebeat start running.
2020-11-16T12:51:55.675Z INFO registrar/registrar.go:97 No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2020-11-16T12:51:55.679Z INFO registrar/registrar.go:134 Loading registrar data from /usr/share/filebeat/data/registry
2020-11-16T12:51:55.679Z INFO registrar/registrar.go:141 States Loaded from registrar: 0
2020-11-16T12:51:55.679Z INFO crawler/crawler.go:72 Loading Inputs: 2
2020-11-16T12:51:55.680Z INFO log/input.go:148 Configured paths: [/var/log/logs-app/app.log]
2020-11-16T12:51:55.680Z INFO input/input.go:114 Starting input of type: log; ID: 14367348903407064248
2020-11-16T12:51:55.680Z INFO log/input.go:148 Configured paths: [/var/log/logs-app/access.log]
2020-11-16T12:51:55.680Z INFO input/input.go:114 Starting input of type: log; ID: 4343579462404243939
2020-11-16T12:51:55.680Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2
2020-11-16T12:51:55.680Z INFO log/harvester.go:255 Harvester started for file: /var/log/logs-app/app.log
2020-11-16T12:51:55.681Z INFO log/harvester.go:255 Harvester started for file: /var/log/logs-app/access.log
2020-11-16T12:51:55.681Z INFO cfgfile/reload.go:150 Config reloader started
2020-11-16T12:51:55.681Z INFO cfgfile/reload.go:205 Loading of config files completed.
2020-11-16T12:51:59.681Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://10.xxx.xx.xx:9200))
2020-11-16T12:51:59.685Z INFO elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.8.1
2020-11-16T12:51:59.691Z INFO template/load.go:128 Template already exists and will not be overwritten.
2020-11-16T12:51:59.691Z INFO instance/beat.go:889 Template successfully loaded.
2020-11-16T12:51:59.691Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://10.xxx.xx.xx:9200)) established
With some other config I can see that the cluster is being queried successfuly
2020-11-16T13:30:36.838Z INFO kubernetes/util.go:86 kubernetes: Using pod name log-deployment-b69f89794-zllpw and namespace demo-monitoring to discover kubernetes node
2020-11-16T13:30:36.853Z INFO kubernetes/util.go:93 kubernetes: Using node ip-10-xxx-xx-xxx.eu-west-1.compute.internal discovered by in cluster pod node query
2020-11-16T13:30:36.853Z INFO kubernetes/watcher.go:182 kubernetes: Performing a resource sync for *v1.PodList
2020-11-16T13:30:36.855Z INFO kubernetes/watcher.go:198 kubernetes: Resource sync done
2020-11-16T13:30:36.856Z INFO kubernetes/watcher.go:242 kubernetes: Watching API for resource events
But still can't see any metadata in the documents