# Use metricbeat to monitor elasticsearch/kibana/beats
**URL:** https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828
**Category:** Elastic Cloud on Kubernetes (ECK)
**Tags:** elastic-stack-monitoring, docker
**Created:** [May 16, 2022, 1:43pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828 "2022-05-16T13:43:04Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 16, 2022, 1:43pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/1 "2022-05-16T13:43:04Z")
Hi i am trying to get metricbeats to monitor the status of my Elasticsearch cluster. If i go to `Stack Monitoring` page it tells me taht metricbeat has not been setup.
I'm confused what im doing wrong here, i don't see any errors in the ECK operator or any of the metricbeat pods.
Metricbeat pods do work as i can see they are pulling in metrics from kubernetes/nodes.
metricbeat.yml
```auto
# metricbeat resources
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: metricbeat
namespace: elastic-system
spec:
type: metricbeat
version: 8.2.0
elasticsearchRef:
name: elasticsearch-prod
config:
metricbeat:
autodiscover:
providers:
- type: kubernetes
scope: cluster
hints.enabled: true
enabled: "true"
node: ${NODE_NAME}
templates:
- condition:
contains:
kubernetes.labels.scrape: es
config:
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- enrich
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
# WARNING: disables TLS as the default certificate is not valid for the pod FQDN
# TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
ssl.verification_mode: "none"
xpack.enabled: true
- condition:
contains:
kubernetes.labels.scrape: kb
config:
- module: kibana
metricsets:
- stats
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
# WARNING: disables TLS as the default certificate is not valid for the pod FQDN
# TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
ssl.verification_mode: "none"
xpack.enabled: true
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
process:
include_top_n:
by_cpu: 5
by_memory: 5
processes:
- .*
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event:
when:
regexp:
system:
filesystem:
mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
- module: kubernetes
period: 10s
node: ${NODE_NAME}
hosts:
- https://${NODE_NAME}:10250
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl:
verification_mode: none
metricsets:
- node
- system
- pod
- container
- volume
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: metricbeat
automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
containers:
- args:
- -e
- -c
- /etc/beat.yml
- -system.hostfs=/hostfs
name: metricbeat
volumeMounts:
- mountPath: /hostfs/sys/fs/cgroup
name: cgroup
- mountPath: /var/run/docker.sock
name: dockersock
- mountPath: /hostfs/proc
name: proc
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MONITORED_ES_USERNAME
value: elastic
- name: MONITORED_ES_PASSWORD
valueFrom:
secretKeyRef:
key: elastic
name: elasticsearch-prod-es-elastic-user
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /sys/fs/cgroup
name: cgroup
- hostPath:
path: /var/run/docker.sock
name: dockersock
- hostPath:
path: /proc
name: proc
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
namespace: elastic-system
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
resources:
- jobs
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
namespace: elastic-system
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: elastic-system
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
---
```
references i ahve been using:
- [Elastic Stack Monitoring with Elastic Cloud on Kubernetes | Elastic Blog](https://www.elastic.co/blog/elastic-stack-monitoring-with-elastic-cloud-on-kubernetes)
- [cloud-on-k8s/stack\_monitoring.yaml at main · elastic/cloud-on-k8s · GitHub](https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/beats/stack_monitoring.yaml)
- [cloud-on-k8s/metricbeat\_hosts.yaml at main · elastic/cloud-on-k8s · GitHub](https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/beats/metricbeat_hosts.yaml)
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 16, 2022, 9:01pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/2 "2022-05-16T21:01:53Z")
Think perhaps you might want to try the `elasticsearch-xpack` module... yeah confusing but think so
> **[Elasticsearch module | Metricbeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-elasticsearch.html#_usage_for_elastic_stack_monitoring_2)**
> ###
>
> Usage for Elastic Stack Monitoring[edit](https://github.com/elastic/beats/edit/8.2/metricbeat/docs/modules/elasticsearch.asciidoc)
>
> The `elasticsearch` module can be used to collect metrics shown in our Stack Monitoring UI in Kibana. To enable this usage, set `xpack.enabled: true` and remove any `metricsets` from the module’s configuration. Alternatively, run `metricbeat modules disable elasticsearch` and `metricbeat modules enable elasticsearch-xpack` .
>
> When this module is used for Elastic Stack Monitoring, it sends metrics to the monitoring index instead of the default index typically used by Metricbeat. For more details about the monitoring index, see [Configuring indices for monitoring](https://www.elastic.co/guide/en/elasticsearch/reference/8.2/config-monitoring-indices.html)./
Hmm maybe some of our docs and sample may need to be updated.
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 27, 2022, 10:26am UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/3 "2022-05-27T10:26:57Z")
yeah i kind of gave up on trying to use metricbeat to monitor my elastic cluster.
---
### Author:  [@framsouza](https://discuss.elastic.co/u/framsouza)
#### Post date: [May 30, 2022, 9:33am UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/4 "2022-05-30T09:33:19Z")
Hey @Kay_Khan ,
Can you please share the output logs of beats? Hopefully there we can have more insights about why you are not getting the monitoring events.
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 31, 2022, 3:33pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/5 "2022-05-31T15:33:11Z")
Is it even possible? From further reading it seems like you need an entirely new seperate elastic stack catered for monitoring, just to monitor the existing "main" stack with metricbeat.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 31, 2022, 4:05pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/6 "2022-05-31T16:05:53Z")
Nope that is a recommended best practice but you can certainly feed monitoring data to the main cluster people do it all the time.
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 31, 2022, 4:09pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/7 "2022-05-31T16:09:27Z")
Okay, if that is the case, i will give it another ago get back to you on this thread with the logs, but i remember there being no errors
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 31, 2022, 4:24pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/8 "2022-05-31T16:24:44Z")
You can see in my config i have `xpack.enabled: true` true on both the Elasticsearch and kibana module. Are you suggesting im missing something else?
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 31, 2022, 4:41pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/9 "2022-05-31T16:41:04Z")
Is there a way to disable self-monitoring if ive already enabled it? ( it makes it difficult to know if stack-monitoring is using self monitored vs metricbeat
// not sure if its this
` # monitoring.kibana.collection.enabled: false`
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 31, 2022, 5:34pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/10 "2022-05-31T17:34:40Z")
Trying this again using slightly different config however im getting an error:
> Exiting: missing field accessing 'metricbeat.modules.3.hosts' (source:'/etc/beat.yml')
followed this documentation - [Elasticsearch module | Metricbeat Reference [master] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-elasticsearch.html)
Tried all syntax's, all with the saem error:
```auto
hosts: []
hosts : ""
hosts :
- ""
```
```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: metricbeat
namespace: elastic-system
spec:
type: metricbeat
version: 8.2.0
elasticsearchRef:
name: elasticsearch-prod
kibanaRef:
name: kibana-prod
config:
setup.template.settings:
index:
routing.allocation.require.type: "monitoring"
metricbeat:
autodiscover:
providers:
- hints:
default_config: {}
enabled: "true"
node: ${NODE_NAME}
type: kubernetes
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
process:
include_top_n:
by_cpu: 5
by_memory: 5
processes:
- .*
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event:
when:
regexp:
system:
filesystem:
mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
- module: kubernetes
period: 10s
node: ${NODE_NAME}
hosts:
- https://${NODE_NAME}:10250
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl:
verification_mode: none
metricsets:
- node
- system
- pod
- container
- volume
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- enrich
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
# WARNING: disables TLS as the default certificate is not valid for the pod FQDN
# TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
ssl.verification_mode: "none"
xpack.enabled: true
- module: kibana
metricsets:
- stats
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
# WARNING: disables TLS as the default certificate is not valid for the pod FQDN
# TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
ssl.verification_mode: "none"
xpack.enabled: true
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: metricbeat
automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
containers:
- args:
- -e
- -c
- /etc/beat.yml
- -system.hostfs=/hostfs
name: metricbeat
volumeMounts:
- mountPath: /hostfs/sys/fs/cgroup
name: cgroup
- mountPath: /var/run/docker.sock
name: dockersock
- mountPath: /hostfs/proc
name: proc
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MONITORED_ES_USERNAME
value: elastic
- name: MONITORED_ES_PASSWORD
valueFrom:
secretKeyRef:
key: elastic
name: elasticsearch-prod-es-elastic-user
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-secret
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-secret
key: access-secret-key
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /sys/fs/cgroup
name: cgroup
- hostPath:
path: /var/run/docker.sock
name: dockersock
- hostPath:
path: /proc
name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
namespace: elastic-system
rules:
- apiGroups:
- ""
resources:
- nodes
- namespaces
- events
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- statefulsets
- deployments
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
namespace: elastic-system
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: elastic-system
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
---
```
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [May 31, 2022, 6:15pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/11 "2022-05-31T18:15:42Z")
I got it working
1. Make sure monitoring collection is enabled:
```auto
PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.collection.enabled": true
}
}
```
1. Disable self monitoring collection for kibana
```auto
config:
monitoring.kibana.collection.enabled: false
```
1. Disable self monitoring collection for Elasticsearch
```auto
PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.elasticsearch.collection.enabled": false
}
}
```
or on each node
```auto
nodeSets:
- name: default
count: 2
config:
xpack.monitoring.elasticsearch.collection.enabled: false
```
**metricbeat.yaml**
```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: metricbeat
namespace: elastic-system
spec:
type: metricbeat
version: 8.2.0
elasticsearchRef:
name: elasticsearch-prod
kibanaRef:
name: kibana-prod
config:
setup.template.settings:
index:
routing.allocation.require.type: "monitoring"
metricbeat:
autodiscover:
providers:
- type: kubernetes
scope: cluster
hints.enabled: true
templates:
- condition:
contains:
kubernetes.labels.scrape: es
config:
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- enrich
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
# WARNING: disables TLS as the default certificate is not valid for the pod FQDN
# TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
ssl.verification_mode: "none"
xpack.enabled: true
- condition:
contains:
kubernetes.labels.scrape: kb
config:
- module: kibana
metricsets:
- stats
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
# WARNING: disables TLS as the default certificate is not valid for the pod FQDN
# TODO: switch this to "certificate" when available: https://github.com/elastic/beats/issues/8164
ssl.verification_mode: "none"
xpack.enabled: true
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
process:
include_top_n:
by_cpu: 5
by_memory: 5
processes:
- .*
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event:
when:
regexp:
system:
filesystem:
mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
- module: kubernetes
period: 10s
node: ${NODE_NAME}
hosts:
- https://${NODE_NAME}:10250
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl:
verification_mode: none
metricsets:
- node
- system
- pod
- container
- volume
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: metricbeat
automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
containers:
- args:
- -e
- -c
- /etc/beat.yml
- -system.hostfs=/hostfs
name: metricbeat
volumeMounts:
- mountPath: /hostfs/sys/fs/cgroup
name: cgroup
- mountPath: /var/run/docker.sock
name: dockersock
- mountPath: /hostfs/proc
name: proc
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MONITORED_ES_USERNAME
value: elastic
- name: MONITORED_ES_PASSWORD
valueFrom:
secretKeyRef:
key: elastic
name: elasticsearch-prod-es-elastic-user
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-secret
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-secret
key: access-secret-key
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /sys/fs/cgroup
name: cgroup
- hostPath:
path: /var/run/docker.sock
name: dockersock
- hostPath:
path: /proc
name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
namespace: elastic-system
rules:
- apiGroups:
- ""
resources:
- nodes
- namespaces
- events
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- statefulsets
- deployments
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
namespace: elastic-system
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: elastic-system
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
---
```
In the previous config, i was putting the modules outside of autodiscover so i was getting errors. Im assuming they should be in autodiscover here because there are multiple nodes in elasicsearch.
1. bit confused by `autodisover` vs `modules` e.g when modules should be udner autodiscover vs not
2. confused what xpack.enabled is doing i read a comment ssaying, so my config is not storing metrics inside of the metricbeat indecies? 😕
```auto
# Set to true to send data collected by module to X-Pack
# Monitoring instead of metricbeat-* indices.
```
1. Do i need those conditions? how do i specify just the config module without specifying the conditions
```auto
templates:
- condition:
contains:
kubernetes.labels.scrape: es
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 1, 2022, 12:01am UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/12 "2022-06-01T00:01:13Z")
> [@Kay\_Khan](#):
>
> bit confused by `autodisover` vs `modules` e.g when modules should be udner autodiscover vs not
That is a bit of a long explanation that I would re-read the [docs](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html)
Be very careful of the scope
`scope`
> (Optional) Specify at what level autodiscover needs to be done at. `scope` can either take `node` or `cluster` as values. `node` scope allows discovery of resources in the specified node. `cluster` scope allows cluster wide discovery. Only `pod` and `node` resources can be discovered at node scope.
But in short when what you are applying modules to something in K8s that can get redeployed or changed then they go under auto discover which is most the time. When I look at your config and yes I see you copied it from one of the sample but it does not make sense to me... .but if it is working great
I always start from the metricbeat [quick start page](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html)
And use [that](https://raw.githubusercontent.com/elastic/beats/8.2/deploy/kubernetes/metricbeat-kubernetes.yaml) .yaml which adds the K8s metricbeat modules at the cluster level then a daemonset (i.e. for each node) the normal metrics
Then I what additional I would need
> [@Kay\_Khan](#):
>
> 1. confused what xpack.enabled is doing i read a comment ssaying, so my config is not storing metrics inside of the metricbeat indecies?
> The `elasticsearch` module can be used to collect metrics shown in our Stack Monitoring UI in Kibana. To enable this usage, set `xpack.enabled: true` and remove any `metricsets` from the module’s configuration.
Basically it redirects the indices to the indices that the Stack Monitoring uses not just the plain ole `metricbeat -*` and put the elasticsearchs monitoring data in something like
`.monitoring-es-7-mb-2022.05.31`
That will get changed to a datastream in 8
BUT if you are doing other monitoring like ordinary CPU / System stuff for your pods that data will go into normal `metribeat-*` indices.. (yup a bit confusing)
And technically you should remove those `metricssets:` ... not sure if our docs or sample is wrong but the docs say as above _"and remove any `metricsets` from the module’s configuration"_
> [@Kay\_Khan](#):
>
> 1. Do i need those conditions? how do i specify just the config module without specifying the conditions
Well in the example it is a label to distinguish the monitoring cluster from the main cluster, i.e. which one to scrape... if you are self monitoring then I would think that is not needed but labels and conditions is how you limit where modules are applied
Hope this helps a bit.. yeah ... its harder than anyone would like, especially at the beginning
---
### Author:  [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)
#### Post date: [June 1, 2022, 7:42am UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/13 "2022-06-01T07:42:51Z")
thank you for the response, you mentioned my config does not make sense to you, could you point out which bit, the majority of this is taken from the samples/recipies. [cloud-on-k8s/config/recipes/beats at main · elastic/cloud-on-k8s · GitHub](https://github.com/elastic/cloud-on-k8s/tree/main/config/recipes/beats) i think a lot of peple including me , rely on these recipes a lot to learn from
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 1, 2022, 2:20pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/14 "2022-06-01T14:20:47Z")
Apologies.
I was speaking of this part of your config but I see that comes from that recipe so it would seem to be valid for collect host / node level metrics .. .at least that is what I think it does.
```auto
...
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
...
```
I use the method from [The Metricbeat Deploying on Kubernetes page](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html#_kubernetes_deploy_manifests) page
That collect the node / host level metrics by deploying metricbeat as a daemonset
> You deploy Metricbeat as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to ensure that there’s a running instance on each node of the cluster. These instances are used to retrieve most metrics from the host, such as system metrics, Docker stats, and metrics from all the services running on top of Kubernetes.
With that in mind I start from that manifest as shown on that page [here](https://raw.githubusercontent.com/elastic/beats/8.2/deploy/kubernetes/metricbeat-kubernetes.yaml)
So perhaps it is 2 different ways to do the same thing... although I believe the daemonset method is the new / more standard / best practice, but perhaps I am wrong or missing something.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 29, 2022, 2:20pm UTC](https://discuss.elastic.co/t/use-metricbeat-to-monitor-elasticsearch-kibana-beats/304828/15 "2022-06-29T14:20:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.