# Docker network at kubernetes

**URL:** <https://discuss.elastic.co/t/docker-network-at-kubernetes/165203>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [January 22, 2019, 9:47am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203 "2019-01-22T09:47:16Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [January 22, 2019, 9:47am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/1 "2019-01-22T09:47:16Z")

</div>

Hello,

I would like to have the network traffic of each container that runs on the Kubernetes cluster retrieved by the metricbeat. I have embed the following configuration (check bold) at the configmap yaml, but metricbeat cannot retrieve network IO.

```
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  #hosts: ["localhost:10250"]
  # If using Red Hat OpenShift remove the previous hosts entry and
  # uncomment these settings:
  hosts: ["https://${HOSTNAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.certificate_authorities:
    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

  ssl.verification_mode: "none"

- module: kubernetes
  enabled: true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_statefulset
    - state_pod
    - state_container
  period: 10s
  hosts: ["kube-state-metrics:8080"]

- module: docker
  enabled: true
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s

```

what do I need to add at the configmap?

thank you.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [January 24, 2019, 3:37am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/2 "2019-01-24T03:37:43Z")

</div>

Hi @hohooss, are you deploying metricbeat into each container?

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [January 24, 2019, 9:48am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/3 "2019-01-24T09:48:21Z")

</div>

I have a cluster of three servers (master,2 x nodes) and metricbeat is deployed (daemonset) on all of them:

```
➜ k get pod -n kube-system -o wide | grep metricbeat
metricbeat-djzgz 1/1 Running 0 43h 172.28.150.24 dev-k8s-node2 <none>
metricbeat-mvlmv 1/1 Running 0 43h 172.28.150.22 dev-k8s-master <none> 
metricbeat-q667f 1/1 Running 0 43h 172.28.150.23 dev-k8s-node1 <none>
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2019, 9:48am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/4 "2019-02-21T09:48:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 18, 2019, 6:14pm UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/5 "2019-03-18T18:14:16Z")

</div>



---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 18, 2019, 6:16pm UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/6 "2019-03-18T18:16:56Z")

</div>

Hi @hohooss,

Is the `network` metricset of the docker module collecting the network metrics you are looking for? They would be collected as `docker.network.*`.

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [March 20, 2019, 9:57am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/7 "2019-03-20T09:57:19Z")

</div>

Hello @jsoriano,

I tried to find the relevant collected info, but I take no results.

 ![Capture1](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60a9797f95cba618f8a1c2c854775b2f6d1c421b.jpeg)

even when I search for docker.\* , I take some of the metrics:  
container  
cpu  
diskio  
info  
memory

any clue why is this happening?

thank you.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 25, 2019, 9:05am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/8 "2019-03-25T09:05:12Z")

</div>

I guess you have the `network` metricset of docker enabled?

Can you see any error in your logs?

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [March 26, 2019, 8:01am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/9 "2019-03-26T08:01:44Z")

</div>

Hello,

below is a log event from one of the metricbeat pods:

```
2019-03-26T07:47:42.154Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1739270,"time":{"ms":52}},"total":{"ticks":11594130,"time":{"ms":309},"value":11594130},"user":{"ticks":9854860,"time":{"ms":257}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":44},"info":{"ephemeral_id":"14ddsaaf-7b6d-4887-b3dd-e5a7997a36a8","uptime":{"ms":927663039}},"memstats":{"gc_next":14725200,"memory_alloc":8041600,"memory_total":845674242912}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":241,"batches":8,"total":241},"read":{"bytes":4248},"write":{"bytes":252911}},"pipeline":{"clients":10,"events":{"active":0,"published":241,"total":241},"queue":{"acked":241}}},"metricbeat":{"docker":{"container":{"events":12,"success":12},"cpu":{"events":12,"success":12},"diskio":{"events":12,"success":12},"info":{"events":1,"success":1},"memory":{"events":12,"success":12}},"kubernetes":{"container":{"events":7,"success":7},"node":{"events":1,"success":1},"pod":{"events":5,"success":5},"state_container":{"events":53,"success":53},"state_deployment":{"events":6,"success":6},"state_node":{"events":6,"success":6},"state_pod":{"events":40,"success":40},"state_replicaset":{"events":6,"success":6},"system":{"events":3,"success":3},"volume":{"events":5,"success":5}},"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":48,"success":48},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":0.08,"15":0.08,"5":0.05,"norm":{"1":0.0025,"15":0.0025,"5":0.0016}}}}}}

```

as you can see, there are network events, but these cannot be parsed to the Kibana.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 26, 2019, 4:09pm UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/10 "2019-03-26T16:09:16Z")

</div>

These are network events from Beats monitoring, not from docker.

Can you see any error related to docker in logs?

If you run `docker stats`, can you see values in the `NET I/O` column?

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [March 27, 2019, 7:08am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/11 "2019-03-27T07:08:26Z")

</div>

NET I/O are all 0s, so there are no stats for docker network.  
what is missing? how to proceed?

The docker version that is installed on systems is the following:

> # docker version  
> Client:  
> Version: 18.06.3-ce  
> API version: 1.38  
> Go version: go1.10.3  
> Git commit: d7080c1  
> Built: Wed Feb 20 02:26:51 2019  
> OS/Arch: linux/amd64  
> Experimental: false  
>   
> Server:  
> Engine:  
> Version: 18.06.3-ce  
> API version: 1.38 (minimum version 1.12)  
> Go version: go1.10.3  
> Git commit: d7080c1  
> Built: Wed Feb 20 02:28:17 2019  
> OS/Arch: linux/amd64  
> Experimental: false

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [April 10, 2019, 7:30am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/12 "2019-04-10T07:30:09Z")

</div>

Hello @jsoriano,

is there anything missing from our docker environment? how can I resolve this issue?

thanks.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 10, 2019, 11:02am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/13 "2019-04-10T11:02:39Z")

</div>

Hi @hohooss,

Sorry for the late reply.

Metricbeat module for docker gets the metrics from docker stats and it seems they are not available there. Could you give more details of your cluster, regarding how you deploy it, and what network plugins you use?

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [April 10, 2019, 12:03pm UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/14 "2019-04-10T12:03:23Z")

</div>

it's kubernetes cluster (v1.13.4) installed with Canal (calico+flannel). Shouldn't Kibana work properly with this kind of setup?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 12, 2019, 5:39pm UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/15 "2019-04-12T17:39:25Z")

</div>

@hohooss we may need to open a new issue to request the support of network stats with this configuration. Could you also send the result of `docker info` in these machines?

Thanks!

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [April 15, 2019, 6:08am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/16 "2019-04-15T06:08:31Z")

</div>

@jsoriano I have appended the docker info from the kubernetes nodes:

```
Containers: 22
 Running: 20
 Paused: 0
 Stopped: 2
Images: 14
Server Version: 18.06.3-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.5.1.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 251.4GiB
Name: < >
ID: < >
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:

```

if you need any further info, please let me know.

thank you.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 16, 2019, 10:03am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/17 "2019-04-16T10:03:47Z")

</div>

@hohooss in order to try to reproduce this, how did you deploy this kubernetes cluster? Is it in bare-metal servers or in cloud?

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [April 16, 2019, 11:16am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/18 "2019-04-16T11:16:26Z")

</div>

@jsoriano, it was deployed in bare-metal (kubeadm).

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 16, 2019, 11:35am UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/19 "2019-04-16T11:35:04Z")

</div>

Thanks @hohooss, I have created an issue for further investigation [https://github.com/elastic/beats/issues/11837](https://github.com/elastic/beats/issues/11837)

---

<div class="post-metadata">

**Author:** ![hohooss](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hohooss](https://discuss.elastic.co/u/hohooss)\
**Post date:** [April 16, 2019, 1:06pm UTC](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203/20 "2019-04-16T13:06:23Z")

</div>

@jsoriano, thank you.

[Next page](https://discuss.elastic.co/t/docker-network-at-kubernetes/165203.md?page=2)