Kibana APM UI not displaying CPU & Memory metric for Kubernetes pods/containers shipping APM data

[ParashB (Paras)]

APM agents are shipping APM metrics to APM server for Kubernetes pods. The kibana UI displays the heap/non-heap memory as well as the thread count but doesn't display the CPU & Memory graphs.

Kibana version: 7.2

Elasticsearch version: 7.2

APM Server version: 7.2.0

APM Agent language and version: Java agent v1.9

Browser version: Chrome

Fresh install or upgraded from other version? Fresh install

APM server is shipping logs to Kafka from where logstash picks it up and ships to Elastic search

The kibana UI displays the heap/non-heap memory as well as the thread count but doesn't display the CPU & Memory graphs. The CPU & memory graps are empty as well as the apm-metrics-* indices do not have CPU & memory field.

Sample APM metrics document:

{
  "_index": "apm-7.2.0-metric-2020.01.22",
  "_type": "_doc",
  "_id": "fgdfsghdd563fgbhv",
  "_version": 1,
  "_score": null,
  "_source": {
    "container": {
      "id": "fdsgfdhgfhd"
    },
    "observer": {
      "hostname": "apm-server-gfhfghvc-fghfghfg-zxjjg",
      "id": "fghfghfghfghfd",
      "type": "apm-server",
      "ephemeral_id": "vcbcv-4f43-6546-5745-tghtghbn56gf",
      "version": "7.2.0",
      "version_major": 7
    },
    "agent": {
      "name": "java",
      "ephemeral_id": "dvbc-vcbvcx-vcx-bvc-vcbvcbcvbfd",
      "version": "1.9.0"
    },
    "process": {
      "pid": 1,
      "title": "/lib/runtime/bin/java"
    },
    "@timestamp": "2020-01-21T18:54:11.151Z",
    "ecs": {
      "version": "1.0.0"
    },
    "service": {
      "name": "gener",
      "runtime": {
        "name": "Java",
        "version": "11"
      },
      "language": {
        "name": "Java",
        "version": "11"
      }
    },
    "host": {
      "hostname": "gener-60-ghgys",
      "os": {
        "platform": "Linux"
      },
      "ip": "10.34.21.61",
      "architecture": "amd64"
    },
    "@version": "1",
    "type": "abc-apm",
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "span": {
      "self_time": {
        "count": 5,
        "sum": {
          "us": 2777
        }
      }
    }
  },
  "fields": {
    "@timestamp": [
      "2020-01-21T18:54:11.151Z"
    ]
  }
}

[Eyal_Koren]

Hi and thanks for the question.

@ParashB

The CPU & memory graps are empty

Are these charts missing, or are they empty?

[ParashB (Paras)]

The graphs exists, but they are empty.

[Eyal_Koren]

@ParashB please set you Java agent's log_level to TRACE, let your application pods run for a few minutes and look in their logs. At this logging level, the agent would print to log anything it is sending to the APM sever, so look for lines that represents the metadata and metrics JSONs. It would look something like this:

2020-01-26 09:18:32.297 [elastic-apm-server-reporter] TRACE co.elastic.apm.agent.report.serialize.DslJsonSerializer - {"metadata":{"service":{"name":"opbeans-java","agent":{"name":"java","ephemeral_id":"bee156da-59c4-45ed-8318-9ee951f64270","version":"1.12.1-SNAPSHOT"},"language":{"name":"Java","version":"10.0.2"},"node":{"configured_name":"MyNode"},"runtime":{"name":"Java","version":"10.0.2"},"version":null},"process":{"pid":1936,"ppid":1841,"title":"/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home/bin/java"},"system":{"architecture":"x86_64","hostname":"Eyals-MBP","platform":"Mac OS X"}}}
...
{"metricset":{"timestamp":1580023102284000,"samples":{"system.process.cpu.total.norm.pct":{"value":0.12859959343492597},"jvm.memory.heap.used":{"value":247145472.0},"jvm.memory.non_heap.used":{"value":119538176.0},"jvm.memory.heap.max":{"value":4294967296.0},"jvm.gc.alloc":{"value":11408352.0},"jvm.memory.non_heap.committed":{"value":123437056.0},"system.cpu.total.norm.pct":{"value":0.23933121905079843},"system.process.memory.size":{"value":9865682944.0},"jvm.memory.heap.committed":{"value":402653184.0},"jvm.memory.non_heap.max":{"value":511705088.0},"system.memory.actual.free":{"value":35700736.0},"system.memory.total":{"value":17179869184.0},"jvm.thread.count":{"value":62.0}}}}
{"metricset":{"timestamp":1580023102284000,"tags":{"name":"G1 Old Generation"},"samples":{"jvm.gc.time":{"value":0.0},"jvm.gc.count":{"value":0.0}}}}
{"metricset":{"timestamp":1580023102284000,"tags":{"name":"G1 Young Generation"},"samples":{"jvm.gc.time":{"value":164.0},"jvm.gc.count":{"value":15.0}}}}

Then paste them please.
Thanks,
Eyal.