# Dial tcp {kubelet\_ip}:10250 i/o timeout while trying to get Metricbeat to talk to Azure Kubernetes Service

**URL:** https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874
**Category:** Beats
**Tags:** metricbeat
**Created:** [May 3, 2022, 8:12pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874 "2022-05-03T20:12:05Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![surprised\_ferret](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@surprised\_ferret](https://discuss.elastic.co/u/surprised_ferret)
#### Post date: [May 3, 2022, 8:12pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/1 "2022-05-03T20:12:05Z")

</div>

Getting these errors when trying to install metricbeat as a daemonset using terraform.

What could be the issue, metricbeat or AKS ?

```auto
2022-05-03T20:09:01.492Z ERROR module/wrapper.go:259 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "{my_aks_url}:10250/stats/summary": dial tcp {my_aks_IP}:10250: i/o timeout

2022-05-03T20:09:01.492Z ERROR module/wrapper.go:259 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get "{my_aks_url}10250/stats/summary": dial tcp {my_aks_IP}:10250: i/o timeout

2022-05-03T20:09:01.612Z ERROR [kubernetes.container] container/container.go:93 error making http request: Get "{my_aks_url}:10250/stats/summary": dial tcp {my_aks_IP}:10250: i/o timeout

```

---

<div class="post-metadata">

### Author: ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)
#### Post date: [May 4, 2022, 4:21pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/2 "2022-05-04T16:21:01Z")

</div>

Hi @surprised_ferret

Are you using the default configuration `hosts: ["https://${NODE_NAME}:10250"]` [beats/metricbeat-kubernetes.yaml at main · elastic/beats · GitHub](https://github.com/elastic/beats/blob/main/deploy/kubernetes/metricbeat-kubernetes.yaml#L120) for this field?

you can try to exec to the one of the metricbeat pods and try to access kubelet:

```auto
kubectl exec -it <metricbeat-pod> bash

token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -k -H "Authorization: Bearer $(echo $token)" "https://${NODE_NAME}:10250/stats/summary"

```

does it fail with the same error?

---

<div class="post-metadata">

### Author: ![surprised\_ferret](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@surprised\_ferret](https://discuss.elastic.co/u/surprised_ferret)
#### Post date: [May 4, 2022, 6:49pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/3 "2022-05-04T18:49:45Z")

</div>

I am passing in this yml to terraform:

```nohighlight
daemonset:
  metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        metricsets:
          - container
          - node
          - pod
          - system
          - volume
        period: 10s
        host: "${NODE_NAME}" # passed in as azurerm_kubernetes_cluster.global.fqdn
        hosts: ["${NODE_NAME}:10250"]
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl.verification_mode: "none"
        processors:
        - add_kubernetes_metadata: ~
      - module: kubernetes
        enabled: true
        metricsets:
          - event
      - module: system
        period: 10s
        metricsets:
          - cpu
          - load
          - memory
          - network
          - process
          - process_summary
        processes: ['.*']
        process.include_top_n:
          by_cpu: 5
          by_memory: 5
      - module: system
        period: 1m
        metricsets:
          - filesystem
          - fsstat
        processors:
        - drop_event.when.regexp:
            system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
      output.elasticsearch:
        protocol: https
        hosts: '${ELASTICSEARCH_HOSTS}' # passed in as ec_deployment.global.elasticsearch.0.https_endpoint
        username: '${ELASTICSEARCH_USERNAME}' # passed in as elasticstack_elasticsearch_security_user.metricbeat_user.username
        password: '${ELASTICSEARCH_PASSWORD}' # passed in as elasticstack_elasticsearch_security_user.metricbeat_user.password
logging.metrics.enabled: false

```

I tried to log into a pod within the metricbeat daemonset, but it says 'not found'.

I also tried to log into a pod within the metricbeat-metricbeat-metrics deployment, but same issue.

---

<div class="post-metadata">

### Author: ![surprised\_ferret](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@surprised\_ferret](https://discuss.elastic.co/u/surprised_ferret)
#### Post date: [May 4, 2022, 7:12pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/4 "2022-05-04T19:12:15Z")

</div>

Actually hold on, I needed to pass in namespace 🙃 hold please...

---

<div class="post-metadata">

### Author: ![surprised\_ferret](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@surprised\_ferret](https://discuss.elastic.co/u/surprised_ferret)
#### Post date: [May 4, 2022, 7:16pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/5 "2022-05-04T19:16:25Z")

</div>

Getting timeout here as well @Tetiana_Kravchenko

```auto
root@metricbeat-metricbeat-78pt9:/usr/share/metricbeat# curl -k -H "Authorization: Bearer $(echo $token)" "https://{my_node_name}.hcp.centralus.azmk8s.io:10250/stats/summary"

curl: (28) Failed to connect to {my_node_name}.hcp.centralus.azmk8s.io port 10250: Connection timed out

```

---

<div class="post-metadata">

### Author: ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)
#### Post date: [May 5, 2022, 12:33pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/6 "2022-05-05T12:33:23Z")

</div>

I think it might be related to the AKS internals - [here](https://docs.microsoft.com/en-us/azure/aks/troubleshooting#im-receiving-tcp-timeouts-such-as-dial-tcp-node_ip10250-io-timeout) is mentioned similar issue in AKS documentation

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 2, 2022, 2:33pm UTC](https://discuss.elastic.co/t/dial-tcp-kubelet-ip-10250-i-o-timeout-while-trying-to-get-metricbeat-to-talk-to-azure-kubernetes-service/303874/7 "2022-06-02T14:33:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
