# Span events missing kubernetes metadata

**URL:** <https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237>\
**Category:** APM\
**Tags:** ruby, server\
**Created:** [February 6, 2020, 6:52pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237 "2020-02-06T18:52:21Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasonmoo-we](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jasonmoo-we](https://discuss.elastic.co/u/jasonmoo-we)\
**Post date:** [February 6, 2020, 6:52pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/1 "2020-02-06T18:52:21Z")

</div>

**APM Server version** : 7.4.2

**APM Agent language and version** : ruby 3.4.0

Hey I'm running a basic apm server sending events to kafka, deployed to kubernetes with a custom helm chart. A rails app has installed the agent gem and is not using any special env vars.

(ruby 3.4.0) -\> (apm server 7.4.2) -\> kafka

When I inspect the events written to the kafka topic, the span events do not contain the kubernetes metadata. We see this metadata for the metrics events.

We tried setting the kubernetes metadata fields using environment variables using the downward api as prescribed in the apm docs, and we verified they are populating in the application container correctly, but this didn't fix the problem.

I have tried capturing the http requests from the agent and they show the k8s metadata in the metadata payload at the start of each post body. So I'm leaning toward this being an issue with the apm server.

Help?

---

<div class="post-metadata">

**Author:** ![jasonmoo-we](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jasonmoo-we](https://discuss.elastic.co/u/jasonmoo-we)\
**Post date:** [February 6, 2020, 7:23pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/2 "2020-02-06T19:23:07Z")

</div>

Update: Tried deploying latest version of apm server 7.5.2 but still not seeing k8s metadata.

```
 {
  "@timestamp": "2020-02-06T19:20:11.105Z",
  "@metadata": {
    "beat": "apm",
    "type": "_doc",
    "version": "7.5.2",
    "pipeline": "apm"
  },
  "trace": {
    "id": "fda55c9bb28f4af2fc986d62944bab91"
  },
  "timestamp": {
    "us": 1581016811105800
  },
  "span": {
    "id": "a2cb061e0c51aec6",
    "subtype": "controller",
    "action": "action",
    "name": "SomeController#dostuff",
    "type": "app",
    "duration": {
      "us": 5959
    }
  },
  "parent": {
    "id": "67879a48e7c98c89"
  },
  "service": {
    "name": "SomeApi",
    "environment": "staging"
  },
  "transaction": {
    "id": "67879a48e7c98c89"
  },
  "observer": {
    "id": "92a48368-83b5-41f6-9f00-7f764031b851",
    "ephemeral_id": "0e2b6815-dd68-4813-a01e-a16bd40797eb",
    "type": "apm-server",
    "hostname": "apm-server-85ff5b5458-pvp9h",
    "version": "7.5.2",
    "version_major": 7
  },
  "ecs": {
    "version": "1.1.0"
  },
  "processor": {
    "name": "transaction",
    "event": "span"
  },
  "agent": {
    "name": "ruby",
    "version": "3.4.0"
  }
}
```

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [February 7, 2020, 10:34am UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/3 "2020-02-07T10:34:15Z")

</div>

Global labels on spans are not supported in the currently released APM Server versions. Support has been implemented though with [PR#2806](https://github.com/elastic/apm-server/pull/2806) and will be part of the next release.

---

<div class="post-metadata">

**Author:** ![jasonmoo-we](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jasonmoo-we](https://discuss.elastic.co/u/jasonmoo-we)\
**Post date:** [February 7, 2020, 6:35pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/4 "2020-02-07T18:35:19Z")

</div>

Cool any eta on that release?

---

<div class="post-metadata">

**Author:** ![jasonmoo-we](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jasonmoo-we](https://discuss.elastic.co/u/jasonmoo-we)\
**Post date:** [February 7, 2020, 10:22pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/5 "2020-02-07T22:22:34Z")

</div>

Update: I pulled down the apm server repo and ran `make release` successfully on the 7.6 branch. I deployed it and didn't see the k8s values in the span.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [February 10, 2020, 9:07am UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/6 "2020-02-10T09:07:02Z")

</div>

@jasonmoo-we sorry, there was a bit of misunderstanding earlier. Kubernetes metadata does not require global labels, and the change that Silvia referred to will not help.

It was a conscious decision to omit Kubernetes metadata from span documents, to keep storage costs down. The idea was that you would be able to use Kubernetes metadata to search for transactions of interest first, and then you would find spans related to a specific transaction.

It might make sense to revisit this decision - perhaps making it optional. What are you intending to do with the metadata?

---

<div class="post-metadata">

**Author:** ![jasonmoo-we](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jasonmoo-we](https://discuss.elastic.co/u/jasonmoo-we)\
**Post date:** [February 10, 2020, 5:54pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/7 "2020-02-10T17:54:23Z")

</div>

Yes ok this makes more sense. I didn't understand the relation to global labels.

We are using the kafka output option on the apm server, and processing events downstream for consumption by a Prometheus metrics stack. It doesn't make sense for us to do some type of metadata decorating across multiple events. Having the k8s metadata is critical for us. I will fork the server and fix it until you guys are able to address the issue. Given that there are multiple output options on the apm server I'd expect the defaults to optimize for elasticsearch, but things like this to be configurable.

Thanks for the prompt response.

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [February 11, 2020, 3:16am UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/8 "2020-02-11T03:16:28Z")

</div>

> [@jasonmoo-we](#):
>
> Given that there are multiple output options on the apm server I'd expect the defaults to optimize for elasticsearch, but things like this to be configurable.

I tend to agree. I've opened an issue to track this: [Make metadata configurable · Issue #3304 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/issues/3304)

---

<div class="post-metadata">

**Author:** ![jasonmoo-we](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jasonmoo-we](https://discuss.elastic.co/u/jasonmoo-we)\
**Post date:** [February 11, 2020, 6:44pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/9 "2020-02-11T18:44:49Z")

</div>

Great! Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2020, 2:45pm UTC](https://discuss.elastic.co/t/span-events-missing-kubernetes-metadata/218237/10 "2020-03-03T14:45:00Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.