Kubernetes autodiscover provider with "hostNetwork: true"

[hamelg]

I want to setup a metricbeat pod to gather host's metrics AND also to collect other metrics in containers via the kubernetes provider.

Exemple :

## Collect apache metrics in apache containers
   metricbeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                  kubernetes.labels.metricbeat: "apache"
              config:
                - module: apache
                  metricsets: ["status"]
                  hosts: ["http://${data.host}:8089"]


## collect k8s nodes statistics
    metricbeat.modules:
    - module: kubernetes
      metricsets:
        - node
        - system
      hosts: ["https://localhost:10250"]

To allow to connect to the node's kubelet (https://localhost:10250), the metricbeat pod is setup with "hostNetwork: true" [1].

When the metricbeat is starting up, it logs the errors :

Querying for pod failed with error: %!(EXTRA string=kubernetes api: Failure 404 pods "myHost.zzz.xx" not found)
Unable to find pod, setting host to localhost

and the provider collects nothing.

a possible workaround is to setup 2 pods :
1 pod to run the kubernetes autodiscover providers with hostNetwork: false
1 pod to run the kubernetes modules with hostNetwork: true.

Is there a mean to run successfully the kubernetes autodiscover provider with "hostNetwork: true" ?

[1] https://kubernetes.io/docs/concepts/policy/pod-security-policy/#host-namespaces

[_flo (Florian Koch)]

Hi,

Have you tested with the ClusterFirstWithHostNet DNS Policy

DNS for Services and Pods - Kubernetes

Regards

[hamelg]

@_flo

ClusterFirstWithHostNet

Unfortunately, it makes no difference.
It seems the issue comes from how the hostname is setup in the pod :
When hostNetwork is true, the pod is setup with the node hostname where the pod is running.
When hostNetwork is false, the pod is setup with the pod name.
So, with hostNetwork:true the kubernetes provider looks up the node to follow with the pod hostname, and it doesn't find it.