Infrastructure UI (kubernetes) - APM Traces Integration

Elastic Observability APM
1

Kibana version: 7.4.2

Elasticsearch version: 7.4.2

APM Server version: 7.4.2

APM Agent language and version: Java agent v1.10.0

Browser version: Chrome 79.0.3945.29

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker 19.03.4

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup?
Kubernetes cluster is OKD 3.10.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I noticed an issue with the kubernetes.pod.uid mismatched if I choose the "View pod APM traces" option from the Infrastructure UI - Kubernetes screen, i.e. the APM traces UI is expecting a pod UID with underscore separator, while the Infrastructure UI is querying with a pod UID using dash/hyphen as the pod UID separator.

Steps to reproduce:

  1. Go to Infrastructure UI - Kubernetes
  2. On one of the pod choose "View pod APM traces"
  3. Redirected to APM traces screen with a sample URL:
    /app/apm#/traces?_g=()&kuery=kubernetes.pod.uid:"7773f7c7-fd30-11e9-be82-005056938be8"

Noticed that the pod.uid is delimited with hyphen. This returned no result on APM traces page. However, If I replaced hyphen with underscore, the APM traces returned the correct pod APM traces, i.e.
/app/apm#/traces?_g=()&kuery=kubernetes.pod.uid:"7773f7c7_fd30_11e9_be82_005056938be8"

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

2

Hi and thanks for reporting.
This is indeed odd. In order to help us pinpoint the problem, please try to provide the following info:

  1. Set the agent's log_level to TRACE. Look for a line starting with: "Found Kubernetes pod UID:" and check what is the discovered pod ID
  2. The agent will log everything sent to APM server. Look for the metadata document, in which see what is the value of the system.kubernetes.pod.uid field
  3. Pick a trace from the APM view that is traced through this pod, use the Action drop down menu to see the transaction document in Discover. Look for the kubernetes.pod.uid value stored in ES
  4. See if using the query bar to filter APM traces based on the pod UID (as in kubernetes.pod.uid : "7773f7c7-fd30-11e9-be82-005056938be8" ) works with the expected UID or the underscore choice

Thanks

3

Hi,

Thank you for your help.

  1. Found Kubernetes pod UID: a428e940_0290_11ea_be82_005056938be8

  2. I could not find this system.kubernetes.pod.uid field. I only found the same field kubernetes.pod.uid : a428e940_0290_11ea_be82_005056938be8.

  3. kubernetes.pod.uid 7773f7c7_fd30_11e9_be82_005056938be8

  4. Matched only with the underscore.

Thank you.

4

Thanks Hendry, this is very helpful.

It may be an OKD-specific thing, where the pod ID used in the cgroup path uses underscores instead of hyphens.

In order to verify what we need to do, two additional requests:

  1. Please upload the contents of such container's /proc/self/cgroup file (or at least one line).
  2. Just to make sure- on your Kubernetes management, you can see the same pod ID but with hyphens?
5

Hi Eyal,

  1. 11:pids:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f268d6d_0293_11ea_be82_005056938be8.slice/docker-ff4285cc2723440d824a79f530607873a0763da610d94f69070ee3c62459af15.scope

  2. From get pod: 3f268d6d-0293-11ea-be82-005056938be8

Thanks.

6

Thanks for verifying!
Please try to download the relevant artefact and verify that with it the integration with Infra UI works as expected.

7

Hi Eyal,

It is working. It is using hyphen instead of underscore now (Discover screen) kubernetes.pod.uid 280a8909-046b-11ea-be82-005056938be8.

Thanks.

1 Like
8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.