# Infrastructure UI (kubernetes) - APM Traces Integration **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [November 8, 2019, 11:17am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096 "2019-11-08T11:17:19Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [November 8, 2019, 11:17am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/1 "2019-11-08T11:17:19Z")
**Kibana version** : 7.4.2 **Elasticsearch version** : 7.4.2 **APM Server version** : 7.4.2 **APM Agent language and version** : Java agent v1.10.0 **Browser version** : Chrome 79.0.3945.29 **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: docker 19.03.4 **Fresh install or upgraded from other version?** Fresh install **Is there anything special in your setup?** Kubernetes cluster is OKD 3.10. **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: I noticed an issue with the kubernetes.pod.uid mismatched if I choose the "View pod APM traces" option from the Infrastructure UI - Kubernetes screen, i.e. the APM traces UI is expecting a pod UID with underscore separator, while the Infrastructure UI is querying with a pod UID using dash/hyphen as the pod UID separator. **Steps to reproduce** : 1. Go to Infrastructure UI - Kubernetes 2. On one of the pod choose "View pod APM traces" 3. Redirected to APM traces screen with a sample URL: /app/apm#/traces?\_g=()&kuery=kubernetes.pod.uid:"7773f7c7-fd30-11e9-be82-005056938be8" Noticed that the pod.uid is delimited with hyphen. This returned no result on APM traces page. However, If I replaced hyphen with underscore, the APM traces returned the correct pod APM traces, i.e. /app/apm#/traces?\_g=()&kuery=kubernetes.pod.uid:"7773f7c7\_fd30\_11e9\_be82\_005056938be8" **Errors in browser console (if relevant)**: **Provide logs and/or server output (if relevant)**: ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 8, 2019, 10:41pm UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/2 "2019-11-08T22:41:24Z")
Hi and thanks for reporting. This is indeed odd. In order to help us pinpoint the problem, please try to provide the following info: 1. Set the agent's [`log_level`](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-level) to `TRACE`. Look for a line starting with: "Found Kubernetes pod UID:" and check what is the discovered pod ID 2. The agent will log everything sent to APM server. Look for the metadata document, in which see what is the value of the `system.kubernetes.pod.uid` field 3. Pick a trace from the APM view that is traced through this pod, use the `Action` drop down menu to see the transaction document in Discover. Look for the `kubernetes.pod.uid` value stored in ES 4. See if using the query bar to filter APM traces based on the pod UID (as in `kubernetes.pod.uid : "7773f7c7-fd30-11e9-be82-005056938be8" `) works with the expected UID or the underscore choice Thanks ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [November 9, 2019, 1:48am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/3 "2019-11-09T01:48:58Z")
Hi, Thank you for your help. 1. Found Kubernetes pod UID: a428e940\_0290\_11ea\_be82\_005056938be8 2. I could not find this `system.kubernetes.pod.uid` field. I only found the same field `kubernetes.pod.uid : a428e940_0290_11ea_be82_005056938be8`. 3. `kubernetes.pod.uid 7773f7c7_fd30_11e9_be82_005056938be8` 4. Matched only with the underscore. Thank you. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 11, 2019, 7:19am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/4 "2019-11-11T07:19:20Z")
Thanks Hendry, this is very helpful. It may be an OKD-specific thing, where the pod ID used in the `cgroup` path uses underscores instead of hyphens. In order to verify what we need to do, two additional requests: 1. Please upload the contents of such container's `/proc/self/cgroup` file (or at least one line). 2. Just to make sure- on your Kubernetes management, you can see the same pod ID but with hyphens? ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [November 11, 2019, 7:45am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/5 "2019-11-11T07:45:51Z")
Hi Eyal, 1. `11:pids:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f268d6d_0293_11ea_be82_005056938be8.slice/docker-ff4285cc2723440d824a79f530607873a0763da610d94f69070ee3c62459af15.scope` 2. From get pod: `3f268d6d-0293-11ea-be82-005056938be8` Thanks. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [November 11, 2019, 9:05am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/6 "2019-11-11T09:05:27Z")
Thanks for verifying! Please try to download [the relevant artefact](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-920/2/artifacts) and verify that with it the integration with Infra UI works as expected. ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [November 11, 2019, 10:15am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/7 "2019-11-11T10:15:43Z")
Hi Eyal, It is working. It is using hyphen instead of underscore now (Discover screen) `kubernetes.pod.uid 280a8909-046b-11ea-be82-005056938be8`. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 2, 2019, 6:15am UTC](https://discuss.elastic.co/t/infrastructure-ui-kubernetes-apm-traces-integration/207096/8 "2019-12-02T06:15:54Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.