Hello,
As my goal, I want to use Jolokia JVM agent with Metricbeat Kubernetes autodiscovery (with Jolokia module) to monitor my Java application statistics.
Usually, we use Deployments to deploy our applications with some HPA's for autoscaling.
And scaling is something that is troublesome for us right now.
We want to get metrics from all Pods of our apps and to be able to distinguish them (to know with which Pod specifically there may be a problem).
For now, we are only able to get the Jolokia module working with autodiscovery Kubernetes Services, but because Services works as LoadBalancers, we are not able to tell, which event is from which Pod (if we have more than 1 replica of the application). Our configuration looks like that:
metricbeat.autodiscover:
providers:
- type: kubernetes
scope: cluster
resource: service
templates:
- condition:
contains:
kubernetes.labels.jolokia-metrics: "enabled"
config:
- module: jolokia
metricsets: ["jmx"]
period: 10s
hosts: ["${data.kubernetes.service.name}.${data.kubernetes.namespace}:9090/jolokia/"]
username: jolokia
password: <redacted>
namespace: ${data.kubernetes.namespace}
jmx.mappings: [...]
My question is, how to set Jolokia module (specifically hosts) when we switch from resource: service to resource: pod?
According to documentation (Autodiscover | Metricbeat Reference [7.16] | Elastic), when we switch to Pod resouce, we get these field:
- kubernetes.container.id
- kubernetes.container.image
- kubernetes.container.name
- kubernetes.namespace
- kubernetes.node.name
- kubernetes.pod.name
- kubernetes.pod.uid
Alas, there isn't one with kubernetes.pod.ip that would solve our problem (Metricbeat with hostNetwork: enabled can easily access pods internal IPs)