Kubernetes metadata missing from APM documents

drinnird · February 5, 2019, 7:49pm

Kibana V. 6.5.4:

ES V. 5.6:

APM V 6.6.0:

APM agent nodeJS v 1.12:

**Installed from docker image docker.elastic.co/apm/apm-server:6.6.0 **:

Fresh install

Is there anything special in your setup? Using kubernetes, deploying APM as a Service, default namespace, no customizations except adding kubernetes metadata:

Sample config:

gist.github.com

https://gist.github.com/digressionary/573939468d9f6d2afe284d2c6014d52b

apm-serverVALUES.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-server-config
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:

This file has been truncated. show original

Problem: Kubernetes metadata is not added to the APM traces:

Steps to reproduce:

Launch APM-server using kubectl and this config file:
https://gist.github.com/digressionary/573939468d9f6d2afe284d2c6014d52b
Configure a node.js app which is running in kubernetes to connect to this APM-server
Hit the node.js app endpoint
the following output is missing kubernetes metadata
https://gist.github.com/digressionary/3ffbca67eb0940cb2992e5475aace8ab

stephenbelanger · February 5, 2019, 11:41pm

v1 of the Node.js agent doesn't report kubernetes data. You'll need to upgrade to v2.

drinnird · February 6, 2019, 7:07pm

Thank you for your kind response.
I upgraded the APM agent to 2.5.1.

All configuration is the same.

However, I am experiencing the same issue (the metadata is not added).

Could this be a problem specific to running kubernetes as "docker-for-desktop" on a MacBook?

Thankyou,

stephenbelanger · February 6, 2019, 7:51pm

Only pod id and name are recorded automatically, and only if they have been found by the container-info module. You can try running that module in your environment to see what it gets, if you like. If that is unable to identify your environment then you will need to supply the data manually via the agent.start(...) options.

drinnird · February 6, 2019, 8:50pm

Okay I'm a bit confused.

Here, the docs list all the kubernetes fields that are "exported" which I took to mean "added to the document"

https://www.elastic.co/guide/en/apm/server/current/exported-fields-kubernetes-processor.html

However, your answer seems to indicate that only two of these fields would be populated, and only in certain circumstances. Could you clarify?

Thank you.

stephenbelanger · February 6, 2019, 8:55pm

That is where data is mapped to in the Elasticsearch documents created by apm-server, if it is given the appropriate input data to include that. It will not record that data itself, it will only map it from the input it gets from agents.

system · February 27, 2019, 4:55pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.