Span events missing kubernetes metadata

Elastic Observability APM
,
1

APM Server version: 7.4.2

APM Agent language and version: ruby 3.4.0

Hey I'm running a basic apm server sending events to kafka, deployed to kubernetes with a custom helm chart. A rails app has installed the agent gem and is not using any special env vars.

(ruby 3.4.0) -> (apm server 7.4.2) -> kafka

When I inspect the events written to the kafka topic, the span events do not contain the kubernetes metadata. We see this metadata for the metrics events.

We tried setting the kubernetes metadata fields using environment variables using the downward api as prescribed in the apm docs, and we verified they are populating in the application container correctly, but this didn't fix the problem.

I have tried capturing the http requests from the agent and they show the k8s metadata in the metadata payload at the start of each post body. So I'm leaning toward this being an issue with the apm server.

Help?

2

Update: Tried deploying latest version of apm server 7.5.2 but still not seeing k8s metadata.

 {
  "@timestamp": "2020-02-06T19:20:11.105Z",
  "@metadata": {
    "beat": "apm",
    "type": "_doc",
    "version": "7.5.2",
    "pipeline": "apm"
  },
  "trace": {
    "id": "fda55c9bb28f4af2fc986d62944bab91"
  },
  "timestamp": {
    "us": 1581016811105800
  },
  "span": {
    "id": "a2cb061e0c51aec6",
    "subtype": "controller",
    "action": "action",
    "name": "SomeController#dostuff",
    "type": "app",
    "duration": {
      "us": 5959
    }
  },
  "parent": {
    "id": "67879a48e7c98c89"
  },
  "service": {
    "name": "SomeApi",
    "environment": "staging"
  },
  "transaction": {
    "id": "67879a48e7c98c89"
  },
  "observer": {
    "id": "92a48368-83b5-41f6-9f00-7f764031b851",
    "ephemeral_id": "0e2b6815-dd68-4813-a01e-a16bd40797eb",
    "type": "apm-server",
    "hostname": "apm-server-85ff5b5458-pvp9h",
    "version": "7.5.2",
    "version_major": 7
  },
  "ecs": {
    "version": "1.1.0"
  },
  "processor": {
    "name": "transaction",
    "event": "span"
  },
  "agent": {
    "name": "ruby",
    "version": "3.4.0"
  }
}
3

Global labels on spans are not supported in the currently released APM Server versions. Support has been implemented though with PR#2806 and will be part of the next release.

4

Cool any eta on that release?

5

Update: I pulled down the apm server repo and ran make release successfully on the 7.6 branch. I deployed it and didn't see the k8s values in the span.

6

@jasonmoo-we sorry, there was a bit of misunderstanding earlier. Kubernetes metadata does not require global labels, and the change that Silvia referred to will not help.

It was a conscious decision to omit Kubernetes metadata from span documents, to keep storage costs down. The idea was that you would be able to use Kubernetes metadata to search for transactions of interest first, and then you would find spans related to a specific transaction.

It might make sense to revisit this decision - perhaps making it optional. What are you intending to do with the metadata?

1 Like
7

Yes ok this makes more sense. I didn't understand the relation to global labels.

We are using the kafka output option on the apm server, and processing events downstream for consumption by a Prometheus metrics stack. It doesn't make sense for us to do some type of metadata decorating across multiple events. Having the k8s metadata is critical for us. I will fork the server and fix it until you guys are able to address the issue. Given that there are multiple output options on the apm server I'd expect the defaults to optimize for elasticsearch, but things like this to be configurable.

Thanks for the prompt response.

8

I tend to agree. I've opened an issue to track this: Make metadata configurable · Issue #3304 · elastic/apm-server · GitHub

1 Like
9

Great! Thanks!

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.