Hi,
we recently updated our Kubernetes version to 1.30.10 (running in AKS) and noticed that metricbeat was starting to get authentication issues after 1 hour.
I found this issue metricbeat runs happily for a while and then starts getting 401 errors when fetching metrics · Issue #42307 · elastic/beats · GitHub and after upgrading metricbeat to 8.15.4 the issue went away.
After this update, however, we noticed that the field kubernetes.deployment.name was no longer being populated in most of our metrics.
I found this issue Fix Metricbeat k8s metadata sometimes not being present at startup by swiatekm · Pull Request #41216 · elastic/beats · GitHub which was included in the release notes for 8.15.4, so I naturally thought this was the issue.
But this did not help - further digging showed that kubernetes.deployment.name was indeed being set for the state_* metrics (like state_deployment), but not in e.g. container or pod metrics.
I have read up on the add_kubernetes_metadata processor, but even with this enabled the field does not get set. And the more I read about this metadata, the less I understand how this could work in the 8.5.1 version we were using before, since we did not use that processor at all.
I have compared the fields in the metricset for 8.5.1 Kubernetes pod metricset | Metricbeat Reference [8.5] | Elastic with the version we upgraded to and kubernetes.deployment.name is not a part of that set. So I'm also really curious were this field get set?
Are there any permission changes that could have caused this? Seems a bit unlikely since we do get the field on all the state_* metrics.
Any input/suggestions would be most welcome.