Use metricbeat to monitor elasticsearch/kibana/beats

Hi i am trying to get metricbeats to monitor the status of my Elasticsearch cluster. If i go to Stack Monitoring page it tells me taht metricbeat has not been setup.

I'm confused what im doing wrong here, i don't see any errors in the ECK operator or any of the metricbeat pods.

Metricbeat pods do work as i can see they are pulling in metrics from kubernetes/nodes.

metricbeat.yml

# metricbeat resources
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat
  namespace: elastic-system
spec:
  type: metricbeat
  version: 8.2.0
  elasticsearchRef:
    name: elasticsearch-prod
  config:
    metricbeat:
      autodiscover:
        providers:
          - type: kubernetes
            scope: cluster
            hints.enabled: true
            enabled: "true"
            node: ${NODE_NAME}
            templates:
              - condition:
                  contains:
                    kubernetes.labels.scrape: es
                config:
                  - module: elasticsearch
                    metricsets:
                      - ccr
                      - cluster_stats
                      - enrich
                      - index
                      - index_recovery
                      - index_summary
                      - ml_job
                      - node_stats
                      - shard
                    period: 10s
                    hosts: "https://${data.host}:${data.ports.https}"
                    username: ${MONITORED_ES_USERNAME}
                    password: ${MONITORED_ES_PASSWORD}
                    # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
                    # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
                    ssl.verification_mode: "none"
                    xpack.enabled: true
              - condition:
                  contains:
                    kubernetes.labels.scrape: kb
                config:
                  - module: kibana
                    metricsets:
                      - stats
                    period: 10s
                    hosts: "https://${data.host}:${data.ports.https}"
                    username: ${MONITORED_ES_USERNAME}
                    password: ${MONITORED_ES_PASSWORD}
                    # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
                    # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
                    ssl.verification_mode: "none"
                    xpack.enabled: true
      modules:
      - module: system
        period: 10s
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        process:
          include_top_n:
            by_cpu: 5
            by_memory: 5
        processes:
        - .*
      - module: system
        period: 1m
        metricsets:
        - filesystem
        - fsstat
        processors:
        - drop_event:
            when:
              regexp:
                system:
                  filesystem:
                    mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
      - module: kubernetes
        period: 10s
        node: ${NODE_NAME}
        hosts:
        - https://${NODE_NAME}:10250
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl:
          verification_mode: none
        metricsets:
        - node
        - system
        - pod
        - container
        - volume
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: metricbeat
        automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
        containers:
        - args:
          - -e
          - -c
          - /etc/beat.yml
          - -system.hostfs=/hostfs
          name: metricbeat
          volumeMounts:
          - mountPath: /hostfs/sys/fs/cgroup
            name: cgroup
          - mountPath: /var/run/docker.sock
            name: dockersock
          - mountPath: /hostfs/proc
            name: proc
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: MONITORED_ES_USERNAME
            value: elastic
          - name: MONITORED_ES_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: elasticsearch-prod-es-elastic-user
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        securityContext:
          runAsUser: 0
        terminationGracePeriodSeconds: 30
        volumes:
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroup
        - hostPath:
            path: /var/run/docker.sock
          name: dockersock
        - hostPath:
            path: /proc
          name: proc
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  namespace: elastic-system
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources:
  - jobs
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: elastic-system 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
  namespace: elastic-system
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---

references i ahve been using:

Think perhaps you might want to try the elasticsearch-xpack module... yeah confusing but think so

Usage for Elastic Stack Monitoringedit

The elasticsearch module can be used to collect metrics shown in our Stack Monitoring UI in Kibana. To enable this usage, set xpack.enabled: true and remove any metricsets from the module’s configuration. Alternatively, run metricbeat modules disable elasticsearch and metricbeat modules enable elasticsearch-xpack .

When this module is used for Elastic Stack Monitoring, it sends metrics to the monitoring index instead of the default index typically used by Metricbeat. For more details about the monitoring index, see Configuring indices for monitoring./

Hmm maybe some of our docs and sample may need to be updated.

yeah i kind of gave up on trying to use metricbeat to monitor my elastic cluster.

Hey @Kay_Khan ,

Can you please share the output logs of beats? Hopefully there we can have more insights about why you are not getting the monitoring events.

Is it even possible? From further reading it seems like you need an entirely new seperate elastic stack catered for monitoring, just to monitor the existing "main" stack with metricbeat.

Nope that is a recommended best practice but you can certainly feed monitoring data to the main cluster people do it all the time.

Okay, if that is the case, i will give it another ago get back to you on this thread with the logs, but i remember there being no errors

You can see in my config i have xpack.enabled: true true on both the Elasticsearch and kibana module. Are you suggesting im missing something else?

Is there a way to disable self-monitoring if ive already enabled it? ( it makes it difficult to know if stack-monitoring is using self monitored vs metricbeat

// not sure if its this

# monitoring.kibana.collection.enabled: false

Trying this again using slightly different config however im getting an error:

Exiting: missing field accessing 'metricbeat.modules.3.hosts' (source:'/etc/beat.yml')

followed this documentation - Elasticsearch module | Metricbeat Reference [master] | Elastic

Tried all syntax's, all with the saem error:

hosts: []
hosts : ""
hosts :
- ""
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat
  namespace: elastic-system
spec:
  type: metricbeat
  version: 8.2.0
  elasticsearchRef:
    name: elasticsearch-prod
  kibanaRef:
    name: kibana-prod
  config:
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"
    metricbeat:
      autodiscover:
        providers:
        - hints:
            default_config: {}
            enabled: "true"
          node: ${NODE_NAME}
          type: kubernetes
      modules:
      - module: system
        period: 10s
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        process:
          include_top_n:
            by_cpu: 5
            by_memory: 5
        processes:
        - .*
      - module: system
        period: 1m
        metricsets:
        - filesystem
        - fsstat
        processors:
        - drop_event:
            when:
              regexp:
                system:
                  filesystem:
                    mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
      - module: kubernetes
        period: 10s
        node: ${NODE_NAME}
        hosts:
        - https://${NODE_NAME}:10250
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl:
          verification_mode: none
        metricsets:
        - node
        - system
        - pod
        - container
        - volume
      - module: elasticsearch
        metricsets:
          - ccr
          - cluster_stats
          - enrich
          - index
          - index_recovery
          - index_summary
          - ml_job
          - node_stats
          - shard
        period: 10s
        hosts: "https://${data.host}:${data.ports.https}"
        username: ${MONITORED_ES_USERNAME}
        password: ${MONITORED_ES_PASSWORD}
        # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
        # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
        ssl.verification_mode: "none"
        xpack.enabled: true
      - module: kibana
        metricsets:
          - stats
        period: 10s
        hosts: "https://${data.host}:${data.ports.https}"
        username: ${MONITORED_ES_USERNAME}
        password: ${MONITORED_ES_PASSWORD}
        # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
        # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
        ssl.verification_mode: "none"
        xpack.enabled: true
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: metricbeat
        automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
        containers:
        - args:
          - -e
          - -c
          - /etc/beat.yml
          - -system.hostfs=/hostfs
          name: metricbeat
          volumeMounts:
          - mountPath: /hostfs/sys/fs/cgroup
            name: cgroup
          - mountPath: /var/run/docker.sock
            name: dockersock
          - mountPath: /hostfs/proc
            name: proc
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: MONITORED_ES_USERNAME
            value: elastic
          - name: MONITORED_ES_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: elasticsearch-prod-es-elastic-user
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: access-secret-key
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        securityContext:
          runAsUser: 0
        terminationGracePeriodSeconds: 30
        volumes:
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroup
        - hostPath:
            path: /var/run/docker.sock
          name: dockersock
        - hostPath:
            path: /proc
          name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  namespace: elastic-system
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
  namespace: elastic-system
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---

I got it working

  1. Make sure monitoring collection is enabled:
PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}
  1. Disable self monitoring collection for kibana
  config:
    monitoring.kibana.collection.enabled: false
  1. Disable self monitoring collection for Elasticsearch
PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.elasticsearch.collection.enabled": false
  }
}

or on each node

  nodeSets:
  - name: default
    count: 2
    config:
      xpack.monitoring.elasticsearch.collection.enabled: false

metricbeat.yaml

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat
  namespace: elastic-system
spec:
  type: metricbeat
  version: 8.2.0
  elasticsearchRef:
    name: elasticsearch-prod
  kibanaRef:
    name: kibana-prod
  config:
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"
    metricbeat:
      autodiscover:
        providers:
          - type: kubernetes
            scope: cluster
            hints.enabled: true
            templates:
              - condition:
                  contains:
                    kubernetes.labels.scrape: es
                config:
                  - module: elasticsearch
                    metricsets:
                      - ccr
                      - cluster_stats
                      - enrich
                      - index
                      - index_recovery
                      - index_summary
                      - ml_job
                      - node_stats
                      - shard
                    period: 10s
                    hosts: "https://${data.host}:${data.ports.https}"
                    username: ${MONITORED_ES_USERNAME}
                    password: ${MONITORED_ES_PASSWORD}
                    # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
                    # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
                    ssl.verification_mode: "none"
                    xpack.enabled: true
              - condition:
                  contains:
                    kubernetes.labels.scrape: kb
                config:
                  - module: kibana
                    metricsets:
                      - stats
                    period: 10s
                    hosts: "https://${data.host}:${data.ports.https}"
                    username: ${MONITORED_ES_USERNAME}
                    password: ${MONITORED_ES_PASSWORD}
                    # WARNING: disables TLS as the default certificate is not valid for the pod FQDN
                    # TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
                    ssl.verification_mode: "none"
                    xpack.enabled: true
      modules:
      - module: system
        period: 10s
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        process:
          include_top_n:
            by_cpu: 5
            by_memory: 5
        processes:
        - .*
      - module: system
        period: 1m
        metricsets:
        - filesystem
        - fsstat
        processors:
        - drop_event:
            when:
              regexp:
                system:
                  filesystem:
                    mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
      - module: kubernetes
        period: 10s
        node: ${NODE_NAME}
        hosts:
        - https://${NODE_NAME}:10250
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl:
          verification_mode: none
        metricsets:
        - node
        - system
        - pod
        - container
        - volume
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: metricbeat
        automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
        containers:
        - args:
          - -e
          - -c
          - /etc/beat.yml
          - -system.hostfs=/hostfs
          name: metricbeat
          volumeMounts:
          - mountPath: /hostfs/sys/fs/cgroup
            name: cgroup
          - mountPath: /var/run/docker.sock
            name: dockersock
          - mountPath: /hostfs/proc
            name: proc
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: MONITORED_ES_USERNAME
            value: elastic
          - name: MONITORED_ES_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: elasticsearch-prod-es-elastic-user
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: aws-secret
                key: access-secret-key
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        securityContext:
          runAsUser: 0
        terminationGracePeriodSeconds: 30
        volumes:
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroup
        - hostPath:
            path: /var/run/docker.sock
          name: dockersock
        - hostPath:
            path: /proc
          name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  namespace: elastic-system
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
  namespace: elastic-system
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---

In the previous config, i was putting the modules outside of autodiscover so i was getting errors. Im assuming they should be in autodiscover here because there are multiple nodes in elasicsearch.

  1. bit confused by autodisover vs modules e.g when modules should be udner autodiscover vs not

  2. confused what xpack.enabled is doing i read a comment ssaying, so my config is not storing metrics inside of the metricbeat indecies? :confused:

  # Set to true to send data collected by module to X-Pack
  # Monitoring instead of metricbeat-* indices.
  1. Do i need those conditions? how do i specify just the config module without specifying the conditions
     templates:
              - condition:
                  contains:
                    kubernetes.labels.scrape: es
1 Like

That is a bit of a long explanation that I would re-read the docs

Be very careful of the scope
scope

(Optional) Specify at what level autodiscover needs to be done at. scope can either take node or cluster as values. node scope allows discovery of resources in the specified node. cluster scope allows cluster wide discovery. Only pod and node resources can be discovered at node scope.

But in short when what you are applying modules to something in K8s that can get redeployed or changed then they go under auto discover which is most the time. When I look at your config and yes I see you copied it from one of the sample but it does not make sense to me... .but if it is working great

I always start from the metricbeat quick start page

And use that .yaml which adds the K8s metricbeat modules at the cluster level then a daemonset (i.e. for each node) the normal metrics

Then I what additional I would need

The elasticsearch module can be used to collect metrics shown in our Stack Monitoring UI in Kibana. To enable this usage, set xpack.enabled: true and remove any metricsets from the module’s configuration.

Basically it redirects the indices to the indices that the Stack Monitoring uses not just the plain ole metricbeat -* and put the elasticsearchs monitoring data in something like

.monitoring-es-7-mb-2022.05.31

That will get changed to a datastream in 8

BUT if you are doing other monitoring like ordinary CPU / System stuff for your pods that data will go into normal metribeat-* indices.. (yup a bit confusing)

And technically you should remove those metricssets: ... not sure if our docs or sample is wrong but the docs say as above "and remove any metricsets from the module’s configuration"

Well in the example it is a label to distinguish the monitoring cluster from the main cluster, i.e. which one to scrape... if you are self monitoring then I would think that is not needed but labels and conditions is how you limit where modules are applied

Hope this helps a bit.. yeah ... its harder than anyone would like, especially at the beginning

thank you for the response, you mentioned my config does not make sense to you, could you point out which bit, the majority of this is taken from the samples/recipies. cloud-on-k8s/config/recipes/beats at main · elastic/cloud-on-k8s · GitHub i think a lot of peple including me , rely on these recipes a lot to learn from

Apologies.

I was speaking of this part of your config but I see that comes from that recipe so it would seem to be valid for collect host / node level metrics .. .at least that is what I think it does.

...
modules:
      - module: system
        period: 10s
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
...

I use the method from The Metricbeat Deploying on Kubernetes page page

That collect the node / host level metrics by deploying metricbeat as a daemonset

You deploy Metricbeat as a DaemonSet to ensure that there’s a running instance on each node of the cluster. These instances are used to retrieve most metrics from the host, such as system metrics, Docker stats, and metrics from all the services running on top of Kubernetes.

With that in mind I start from that manifest as shown on that page here

So perhaps it is 2 different ways to do the same thing... although I believe the daemonset method is the new / more standard / best practice, but perhaps I am wrong or missing something.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.