APM Server version: 7.4.2
APM Agent language and version: ruby 3.4.0
Hey I'm running a basic apm server sending events to kafka, deployed to kubernetes with a custom helm chart. A rails app has installed the agent gem and is not using any special env vars.
(ruby 3.4.0) -> (apm server 7.4.2) -> kafka
When I inspect the events written to the kafka topic, the span events do not contain the kubernetes metadata. We see this metadata for the metrics events.
We tried setting the kubernetes metadata fields using environment variables using the downward api as prescribed in the apm docs, and we verified they are populating in the application container correctly, but this didn't fix the problem.
I have tried capturing the http requests from the agent and they show the k8s metadata in the metadata payload at the start of each post body. So I'm leaning toward this being an issue with the apm server.