Kubernetes labels are missing in pod and container metricsets

Elastic Stack Beats
1

Hi,

I am currently evaluating different approaches to filter filter for deployments / statefulsets and show the child pods and containers including their metrics.

I noticed that in all metricsets having the prefix state_ the kubernetes labels are shown like this:

kubernetes.labels.appl:	redis
kubernetes.labels.cluster: poc
kubernetes.labels.part-of: elasticStack

In container and pod events (without stat_ prefix, these labels are stored in a different manner:

kubernetes.container._module.labels.appl: redis
kubernetes.container._module.labels.cluster: poc
kubernetes.container._module.labels.part-of: elasticStack

For pods they are stored below kubernetes.pod._module.labels.*.

Could you please tell me:

  1. why did you choose different fields for storing the same logical data?
  2. Is there any method available in kibana to filter for joining / combining both fields to a container? Creating a manual filter with an ES-Query where I query both fields with or condition is not an option. That is unable to be used by normal users with dynamically changing filter queries.
  3. Is there any option in metricbeat NOT to add ._module to the labels? I would like to see all kubernetes labels in kubernetes.labels - regardless of the module which captured it / which kubernetes resource is affected.
  4. If currently no solution is available my only idea is writing a logstash ruby filter which will dynamically clone the the field kubernetes.<module>._module.labels.* to kubernetes.labels.*

I am open for any solutions. So hopefully I have just missed some easy option, because I don't think that my usecase is so special.

thanks and regards, Andreas

2

Hi @asp,

unfortunately I'm unable to tell you why this is designed as is, I'm leaving that question open so some of the beats/metricbeats members can clarify. I also wonder why _module is there and if underscore has any special meaning.

I opened an issue I hope will be working if soon if I get no comments preventing it, that would do a more homogeneous use of labels reporting path and some pre-processing filtering: https://github.com/elastic/beats/issues/12938

In the mean time, I only can point you to the drop fields processor to solve 3.: https://www.elastic.co/guide/en/beats/metricbeat/current/drop-fields.html

I haven't tested it, and the documentation doesn't indicate if the fields need to be exact path, can be partials, or if wildcards are allowed ... hope that helps anyway

3

@pmercado thanks for your reply.

I am using following workaround now:
ship all metricbeat probes to logstash and copy all fields below _module to kubernetes root.

Details can be found here. It is not as generic as it should, but it is the fastest workaround which works for me.

4

This is bug introduced earlier and just got fixed with https://github.com/elastic/beats/pull/13433