Problems to deploy Metricbeat in Kubernetes

Elastic Stack Beats
1

Hello, there,

I have a problem deploying a Metricbeat in Kubernetes. I use the following documentation: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html

But when I deploy a Metricbeat, my output configuration in the container looks like this:

output.elasticsearch:
  hosts: ['elasticsearch:9200']
  username: elastic
  password: changeme

My configuration in the YAML file looks like this:

    - name: ELASTICSEARCH_HOST
      value: 10.x.x.x
    - name: ELASTICSEARCH_PORT
      value: "9200"
    #- name: ELASTICSEARCH_USERNAME
    # value: elastic
    #- Name: ELASTICSEARCH_PASSWORD
    # value: changeme

I don't understand why the changes in the YAML file have no effect on the output.elasticsearch configuration.

Does anyone have an idea?

Best Regards

2

Hi @Artur_Becker,

Notice that there are two places where you need to edit environment variables, one for the DaemonSet and another one for the Deployment, is it possible that you only changed one?

3

Hi @jsoriano,

I have adjusted the configuration in both places. I post the whole configuration file.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    # To enable hints based autodiscover uncomment this:
    metricbeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      host: ${NODE_NAME}
    #      hints.enabled: true

    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          in_cluster: true

    #cloud.id: ${ELASTIC_CLOUD_ID}
    #cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['10.x.x.x:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["localhost:10255"]
---
# Deploy a Metricbeat instance per node for node metrics retrieval
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      serviceAccountName: metricbeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.4.2
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
          "-system.hostfs=/hostfs",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: "10.x.x.x"
        - name: ELASTICSEARCH_PORT
          value: "9200"
        #- name: ELASTICSEARCH_USERNAME
        #  value: elastic
        #- name: ELASTICSEARCH_PASSWORD
        #  value: changeme
        #- name: ELASTIC_CLOUD_ID
        #  value:
        #- name: ELASTIC_CLOUD_AUTH
        #  value:
        - name: NODE_NAME
          valueFrom:
             ...                                                             # I had to cut this part to be able to post my answer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata:

    #cloud.id: ${ELASTIC_CLOUD_ID}
    #cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['10.x.x.x:9200']
---
apiVersion: v1
...                                                             # I had to cut this part to be able to post my answer
---
# Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      serviceAccountName: metricbeat
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.4.2
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: "10.x.x.x"
        - name: ELASTICSEARCH_PORT
          value: "9200"
        #- name: ELASTICSEARCH_USERNAME
        #  value: elastic
        #- name: ELASTICSEARCH_PASSWORD
        #  value: changeme
        #- name: ELASTIC_CLOUD_ID
        #  value:
        #- name: ELASTIC_CLOUD_AUTH
        #  value:
        - name: NODE_NAME
          valueFrom:
            ...                                                             # I had to cut this part to be able to post my answer

It looks like the default values are always taken in the Metricbeat container configuration file for the output section and the configuration I customized is ignored.

Best Regards

4

In this config it seems that you have modified the output configuration:

But you only need to modify the environment variables, they will get resolved here on runtime.

5

I have adjusted the configuration now, but the error still persists.

env | grep ELASTIC
ELASTIC_CONTAINER=true
ELASTICSEARCH_HOST=10.21.253.153
ELASTICSEARCH_PORT=9200

The environment variables looks good for me.

The metricbeat.yml file:

 cat metricbeat.yml
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
- add_cloud_metadata:

output.elasticsearch:
  hosts: ['elasticsearch:9200']
  username: elastic
  password: changeme

The metricbeat-kubernetes.yaml file:

output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']

It is interesting to note that Username and Password are also configured, although I did not set these values in output.elasticsearch and commented out the environment variables for them.

Best Regards

6

Inside the container, in /etc/metricbeat.yml you should see the same as in the metricbeat-kubernetes.yaml:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
7

Unfortunately not. The configuration file looks like this:

  output.elasticsearch:
  hosts: ['elasticsearch:9200']
  username: elastic
  password: changeme

Because the wrong server is set in the configuration, the metric data are also not sent to the Elasticsearch DB, I guess. This frustrates me a little bit, because I don't know what exactly went wrong.

With your help I were able to fix the metricbeat-kubernetes.yaml. Also the environment variables within the container are correct.

So why does the configuration within the container use the default values? And even password and username, where these values are not set within metricbeat-kubernetes.yaml.

Best Regards

8

So why does the configuration within the container use the default values?

This is a good question, could you double-check that you haven't done more modifications to the metricbeat-kubernetes.yaml file apart of the environment variables? Could you download again the original and compare?
The output in the config maps looks like this in the original file:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

If you are deploying this file directly to kubernetes with kubectl or similar, the configmap is placed directly in the container as metricbeat.yml, without any modification. Environment variables in this file are then replaced by metricbeat on runtime, but never written to the file.

Are you using kubectl or you use some other tool that could have replaced the environment variables at some moment?

9

Hi Jaime,

I downloaded the metricbeat-kubernetes.yaml again and only changed the configuration of the environment variables in both places.

Then I deployed the containers inside Kubernetes. For this I used the following command:

kubectl create -f ./metricbeat-kubernetes.yaml

But the result is the same. I will build a new Kubernetes instance and try to reproduce the problem there.

Greetings

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.