Metricbeat 7.4.0 and Kubernetes API - Issue

[delphi (Luis Mauricio)]

Hi, there.

Based on GitHub Beats Issue, I'm getting an problem related with k8s API, even using 1.13 clusters. After downgraded Metricbeat to 7.3.2, it was solved.

My config is:

metricbeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false

    queue.mem:
      events: 4096

    setup.dashboards:
      enabled: true
      index: "mtb-k8s-*"

    setup.template:
      enabled: true
      overwrite: true
      name: mtb-k8s
      pattern: "mtb-k8s-*"
      settings.index:
        number_of_shards: 3
        number_of_replicas: 0
        number_of_routing_shards: 30
        routing.allocation.require.node_type: "hot"

    monitoring.enabled: "true"
    monitoring.elasticsearch.username: ${beats-username}
    monitoring.elasticsearch.password: ${beats-password}

    setup.ilm:
      enabled: false

    output.elasticsearch:
      worker: 2
      hosts: http://asdasdasdasd:9200
      username: ${metricbeat-elastic-username}
      password: ${metricbeat-elastic-password}
      indices:
        - index: "mtb-k8s-%{[kubernetes.namespace]}"
          when.has_fields: kubernetes.namespace
        - index: "mtb-k8s-node-metrics"
          when.contains:
            metricset.name: "state_node"

    setup.kibana:
      host: "https://asdasdasd:443"

    metricbeat.modules:
      kubernetes:
      enabled: true
      config:
        - module: kubernetes
          metricsets:
            - state_node
            - state_deployment
            - state_replicaset
            - state_statefulset
            - state_pod
            - state_container
            # - state_cronjob - After the downgrade
          add_metadata: true
          labels.dedot: true
          annotations.dedot: true
          period: 10s
          host: ${NODE_NAME}
          #Even pointing to a newest kube-state-metrics service, the issue still exists.
          hosts: ["asdfasfasfasf.svc:8080"]

I hope you can fix that.

[pmercado (PM)]

Hi @delphi,

I'm not able to reproduce this case.
Can you provide the apiGroups for the loads you are measuring?

I'm using kubernetes 1.13.10, and testing deployments and statefulsets. No issues so far.

[pmercado (PM)]

@delphi

saw the issue, thanks a lot for reporting it, getting back to the GH thread.

[delphi (Luis Mauricio)]

Ok, @pmercado.

Thanks.