While migrating from filebeat 7.6.0 to 7.10.x we discovered that we can not pull kubernetes namespace annotations into events anymore , here is the minimal example config:
filebeat.yml: |
filebeat.autodiscover:
providers:
- add_resource_metadata:
namespace:
enabled: true
include_annotations:
- test
node:
enabled: true
annotations.dedot: true
hints.enabled: true
hints.default_config:
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
type: container
include_annotations:
- test
labels.dedot: true
type: kubernetes
filebeat.registry.flush: 5s
http.enabled: true
http.port: 5066
name: ${NODE_NAME}
output.console:
pretty: true
processors:
- drop_event:
when:
not:
equals:
kubernetes.namespace: "test"
namespace has one pod deployed and annotation test assigned, also pod itself annotated with test - don't be confused - we are expecting to have both pod and namespace annotations to be available in the event fields
here is the kubernetes metadata we are getting with filebeat 7.6.0:
"kubernetes": {
"container": {
"image": "chentex/random-logger:latest",
"name": "random-logger"
},
"node": {
"labels": {
"minikube_k8s_io/commit": "23f40a012abb52eff365ff99a709501a61ac5876",
"kubernetes_io/arch": "amd64",
"kubernetes_io/hostname": "minikube",
"minikube_k8s_io/updated_at": "2021_01_21T16_33_18_0700",
"minikube_k8s_io/version": "v1.15.1",
"beta_kubernetes_io/os": "linux",
"dc": "test",
"beta_kubernetes_io/arch": "amd64",
"kubernetes_io/os": "linux",
"minikube_k8s_io/name": "minikube",
"node-role_kubernetes_io/master": ""
},
"name": "minikube",
"uid": "ad169c1a-fd40-447d-9ac0-8db55ee64aa4"
},
"namespace_uid": "8094f717-eda1-49d7-9680-872248fcd8d4",
"namespace_labels": {
"testlabel": "234"
},
"namespace_annotations": {
"test": "test"
},
"pod": {
"labels": {
"run": "random-logger"
},
"annotations": {
"test": "test"
},
"name": "random-logger",
"uid": "c9129cbd-effa-48a8-a8a0-24d321c7ad7f"
},
"namespace": "test"
},
after upgrade to 7.10.1 or 7.10.2 - it looks different (namespace_annotations is gone)
"kubernetes": {
"namespace_uid": "8094f717-eda1-49d7-9680-872248fcd8d4",
"namespace_labels": {
"testlabel": "234"
},
"pod": {
"uid": "c9129cbd-effa-48a8-a8a0-24d321c7ad7f",
"name": "random-logger"
},
"namespace": "test",
"labels": {
"run": "random-logger"
},
"annotations": {
"test": "test"
},
"container": {
"name": "random-logger",
"image": "chentex/random-logger:latest"
},
"node": {
"name": "minikube",
"uid": "ad169c1a-fd40-447d-9ac0-8db55ee64aa4",
"labels": {
"kubernetes_io/arch": "amd64",
"kubernetes_io/hostname": "minikube",
"dc": "test",
"kubernetes_io/os": "linux",
"node-role_kubernetes_io/master": "",
"minikube_k8s_io/updated_at": "2021_01_21T16_33_18_0700",
"minikube_k8s_io/commit": "23f40a012abb52eff365ff99a709501a61ac5876",
"beta_kubernetes_io/os": "linux",
"beta_kubernetes_io/arch": "amd64",
"minikube_k8s_io/version": "v1.15.1",
"minikube_k8s_io/name": "minikube"
}
}
},
is it expected behaviour? Can it be fixed?
Update: looks like 7.6.1 was the last version with working namespace_annotations