Hi,
I'm trying to run Journalbeat with processor 'add_kubernetes_metadata' to get the metadata of the kubernetes cluster.
Journalbeat is running on the kubernetes nodes started by a daemonset with the following configuration:
journalbeat.inputs:
- paths: ["/var/log/journal"]
seek: cursor
processors:
- add_cloud_metadata: ~
- add_kubernetes_metadata:
in_cluster: true
add_resource_metadata:
default_indexers.enabled: false
default_matchers.enabled: false
indexers:
- container:
matchers:
- fields:
lookup_fields: ["container.id"]
output.elasticsearch:
hosts: ['${ELASTICSEARCHHOSTS}']
protocol: 'https'
username: 'elastic'
password: "${ELASTICPASSWORD}"
ssl.certificate_authorities: ["${CA}"]
logging.level: debug
logging.selectors: [ "kubernetes" ]
In the log output I see following:
2021-07-23T08:20:27.134Z INFO instance/beat.go:665 Home path: [/usr/share/journalbeat] Config path: [/usr/share/journalbeat] Data path: [/usr/share/journalbeat/data]
Logs path: [/usr/share/journalbeat/logs]
2021-07-23T08:20:27.139Z INFO instance/beat.go:673 Beat ID: fe15249c-843e-4513-8113-a79bc4f45f2a
2021-07-23T08:20:27.156Z INFO add_kubernetes_metadata/kubernetes.go:71 add_kubernetes_metadata: kubernetes env detected, with version: v1.20.0
2021-07-23T08:20:27.157Z INFO [kubernetes] kubernetes/util.go:114 kubernetes: Using pod name journalbeat-7kkgq and namespace beats to discover kubernetes node {"l
ibbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.178Z INFO [kubernetes] kubernetes/util.go:120 kubernetes: Using node kube-ingress-1 discovered by in cluster pod node query {"libbeat.processor
": "add_kubernetes_metadata"}
2021-07-23T08:20:27.178Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:162 Initializing a new Kubernetes watcher using host: kube-ingress-1 {"l
ibbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.279Z DEBUG [kubernetes] kubernetes/watcher.go:184 cache sync done
2021-07-23T08:20:27.380Z DEBUG [kubernetes] kubernetes/watcher.go:184 cache sync done
2021-07-23T08:20:27.481Z DEBUG [kubernetes] kubernetes/watcher.go:184 cache sync done
2021-07-23T08:20:27.481Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: beats/filebeat-l5xjt {"libbeat.processor": "add_
kubernetes_metadata"}
2021-07-23T08:20:27.482Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 08c4de66ba9ec36de1b1b6fca1b2c1c738f46eeed0be55d7db52142071bb9
a04 for pod beats/filebeat-l5xjt {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.482Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: beats/journalbeat-7kkgq {"libbeat.processor": "add_
kubernetes_metadata"}
2021-07-23T08:20:27.482Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index d1706896f221b36af7f0f178e8232d416f08dab5b0aaba0653e8f3432fb5b
1aa for pod beats/journalbeat-7kkgq {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.482Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/kube-proxy-v4hdk {"libbeat.processor
": "add_kubernetes_metadata"}
2021-07-23T08:20:27.482Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 8682e265065634ba6e0dbfe7955a588183d82b0d3985fb8b997d2d6fd6ce1
9f2 for pod kube-system/kube-proxy-v4hdk {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.482Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/nginx-proxy-kube-ingress-1 {"libbeat.p
rocessor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 288b69d078340dcc6fa65b2ab548e35c805e3a52d4bd0a2b016414c22f9bc
e6d for pod kube-system/nginx-proxy-kube-ingress-1 {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/calico-node-ptj2t {"libbeat.processor
": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 2a65233371a7dfcd24154b49839ba891875bc749af3c8e35913e1f91b0f8f
645 for pod kube-system/calico-node-ptj2t {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index aef6580dbb2786d1975c607f6f03d5c2488c754d9c1e109b34fcb2fd4609f
e13 for pod kube-system/calico-node-ptj2t {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 18b6d800bcf26e8a4603cab4fdf5397700dcaeb68eb3da2709f199eeb064d
674 for pod kube-system/calico-node-ptj2t {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: ingress-nginx/ingress-nginx-controller-jsq9x {"l
ibbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 998933c97f61a8fc74ba4137c5888be82313dd25f2f344721bb28764c33ff
92f for pod ingress-nginx/ingress-nginx-controller-jsq9x {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/nodelocaldns-znrvz {"libbeat.processor
": "add_kubernetes_metadata"}
2021-07-23T08:20:27.483Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:298 Created index 3fdc04440f6806c94e86081939bd32d67adb5aaa4e1ac54406166dc7b73d799e for pod kube-system/nodelocaldns-znrvz {"libbeat.processor": "add_kubernetes_metadata"}
2021-07-23T08:20:30.142Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2021-07-23T08:20:30.143Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-07-23T08:20:30.144Z INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/journalbeat", "data": "/usr/share/
journalbeat/data", "home": "/usr/share/journalbeat", "logs": "/usr/share/journalbeat/logs"}, "type": "journalbeat", "uuid": "fe15249c-843e-4513-8113-a79bc4f45f2a"}}}
2021-07-23T08:20:30.144Z INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "3ddad4cee7394d1643023604f246cd5ab6d8cfbb", "libbeat":
"7.13.3", "time": "2021-07-02T11:17:29.000Z", "version": "7.13.3"}}}
2021-07-23T08:20:30.144Z INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.15.13"}}}
2021-07-23T08:20:30.147Z INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-07-22T13:39:43Z","conta
inerized":true,"name":"journalbeat-7kkgq","ip":["127.0.0.1/8","::1/128","10.233.98.7/32","fe80::487d:92ff:fe9a:a24c/64"],"kernel_version":"5.4.0-73-generic","mac":["4a:7d:92:9a:a2
:4c"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","t
imezone_offset_sec":0,"id":"8101459f96f52aa8d2c2bf8a41b09ad4"}}}
2021-07-23T08:20:30.148Z INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner
","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kil
l","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","se
tuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap",
"net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/journalbeat", "exe": "/usr/share/journalbeat/journalbeat", "name": "
journalbeat", "pid": 8, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-07-23T08:20:26.500Z"}}}
2021-07-23T08:20:30.149Z INFO instance/beat.go:309 Setup Beat: journalbeat; Version: 7.13.3
2021-07-23T08:20:30.149Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'journalbeat-7.13.3' as ILM is enabled.
2021-07-23T08:20:30.150Z WARN [cfgwarn] tlscommon/config.go:105 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alte
rnative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-07-23T08:20:30.151Z INFO eslegclient/connection.go:99 elasticsearch url: https:<ELASTICSEARCHHOST>:9200
2021-07-23T08:20:30.152Z INFO [publisher] pipeline/module.go:113 Beat name: journalbeat-7kkgq
2021-07-23T08:20:30.152Z WARN [cfgwarn] beater/journalbeat.go:55 EXPERIMENTAL: Journalbeat is experimental.
2021-07-23T08:20:30.167Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-07-23T08:20:30.167Z INFO instance/beat.go:473 journalbeat start running.
2021-07-23T08:20:30.168Z INFO [journalbeat] beater/journalbeat.go:93 journalbeat is running! Hit CTRL-C to stop it.
2021-07-23T08:20:30.169Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:248 No container match string, not adding kubernetes data {"libbeat.processor
": "add_kubernetes_metadata"}
2021-07-23T08:20:30.169Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:248 No container match string, not adding kubernetes data {"libbeat.processor
": "add_kubernetes_metadata"}
2021-07-23T08:20:30.170Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:248 No container match string, not adding kubernetes data {"libbeat.processor
": "add_kubernetes_metadata"}
What's the reason for the message 'No container match string ...' ?
How can I solve this problem to get the kubernetes metadata ?
Thnx for any hints !
Detlef
