Docker network at kubernetes

Elastic Stack Beats
1

Hello,

I would like to have the network traffic of each container that runs on the Kubernetes cluster retrieved by the metricbeat. I have embed the following configuration (check bold) at the configmap yaml, but metricbeat cannot retrieve network IO.

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  #hosts: ["localhost:10250"]
  # If using Red Hat OpenShift remove the previous hosts entry and
  # uncomment these settings:
  hosts: ["https://${HOSTNAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.certificate_authorities:
    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

  ssl.verification_mode: "none"

- module: kubernetes
  enabled: true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_statefulset
    - state_pod
    - state_container
  period: 10s
  hosts: ["kube-state-metrics:8080"]

- module: docker
  enabled: true
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s

what do I need to add at the configmap?

thank you.

2

Hi @hohooss, are you deploying metricbeat into each container?

3

I have a cluster of three servers (master,2 x nodes) and metricbeat is deployed (daemonset) on all of them:

➜ k get pod -n kube-system -o wide | grep metricbeat
metricbeat-djzgz                         1/1     Running   0          43h     172.28.150.24   dev-k8s-node2    <none>
metricbeat-mvlmv                         1/1     Running   0          43h     172.28.150.22   dev-k8s-master   <none> 
metricbeat-q667f                         1/1     Running   0          43h     172.28.150.23   dev-k8s-node1    <none>
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

6

Hi @hohooss,

Is the network metricset of the docker module collecting the network metrics you are looking for? They would be collected as docker.network.*.

7

Hello @jsoriano,

I tried to find the relevant collected info, but I take no results.

Capture1
Capture1.JPG2211×755 128 KB

even when I search for docker.* , I take some of the metrics:
container
cpu
diskio
info
memory

any clue why is this happening?

thank you.

8

I guess you have the network metricset of docker enabled?

Can you see any error in your logs?

9

Hello,

below is a log event from one of the metricbeat pods:

2019-03-26T07:47:42.154Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1739270,"time":{"ms":52}},"total":{"ticks":11594130,"time":{"ms":309},"value":11594130},"user":{"ticks":9854860,"time":{"ms":257}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":44},"info":{"ephemeral_id":"14ddsaaf-7b6d-4887-b3dd-e5a7997a36a8","uptime":{"ms":927663039}},"memstats":{"gc_next":14725200,"memory_alloc":8041600,"memory_total":845674242912}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":241,"batches":8,"total":241},"read":{"bytes":4248},"write":{"bytes":252911}},"pipeline":{"clients":10,"events":{"active":0,"published":241,"total":241},"queue":{"acked":241}}},"metricbeat":{"docker":{"container":{"events":12,"success":12},"cpu":{"events":12,"success":12},"diskio":{"events":12,"success":12},"info":{"events":1,"success":1},"memory":{"events":12,"success":12}},"kubernetes":{"container":{"events":7,"success":7},"node":{"events":1,"success":1},"pod":{"events":5,"success":5},"state_container":{"events":53,"success":53},"state_deployment":{"events":6,"success":6},"state_node":{"events":6,"success":6},"state_pod":{"events":40,"success":40},"state_replicaset":{"events":6,"success":6},"system":{"events":3,"success":3},"volume":{"events":5,"success":5}},"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":48,"success":48},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":0.08,"15":0.08,"5":0.05,"norm":{"1":0.0025,"15":0.0025,"5":0.0016}}}}}}

as you can see, there are network events, but these cannot be parsed to the Kibana.

10

These are network events from Beats monitoring, not from docker.

Can you see any error related to docker in logs?

If you run docker stats, can you see values in the NET I/O column?

11

NET I/O are all 0s, so there are no stats for docker network.
what is missing? how to proceed?

The docker version that is installed on systems is the following:

# docker version
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:51 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:28:17 2019
OS/Arch: linux/amd64
Experimental: false

12

Hello @jsoriano,

is there anything missing from our docker environment? how can I resolve this issue?

thanks.

13

Hi @hohooss,

Sorry for the late reply.

Metricbeat module for docker gets the metrics from docker stats and it seems they are not available there. Could you give more details of your cluster, regarding how you deploy it, and what network plugins you use?

14

it's kubernetes cluster (v1.13.4) installed with Canal (calico+flannel). Shouldn't Kibana work properly with this kind of setup?

15

@hohooss we may need to open a new issue to request the support of network stats with this configuration. Could you also send the result of docker info in these machines?

Thanks!

16

@jsoriano I have appended the docker info from the kubernetes nodes:

Containers: 22
 Running: 20
 Paused: 0
 Stopped: 2
Images: 14
Server Version: 18.06.3-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.5.1.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 251.4GiB
Name: < >
ID: < >
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:

if you need any further info, please let me know.

thank you.

17

@hohooss in order to try to reproduce this, how did you deploy this kubernetes cluster? Is it in bare-metal servers or in cloud?

18

@jsoriano, it was deployed in bare-metal (kubeadm).

19

Thanks @hohooss, I have created an issue for further investigation https://github.com/elastic/beats/issues/11837

20

@jsoriano, thank you.