I'm trying to use kuberntes autodiscover templating to collect metrics from a subset of pods, but I am not able to get output unless I remove the template and set hint.enabled=true
// metricbeat.yaml
providers:
- type: kubernetes
host: ${HOSTNAME} // also tried ${NODE_NAME}
include_annotations: ["metribeat"]
templates:
- condition.contains:
kubernetes.annotations.metribeat: "true"
config:
- module: kubernetes
metricsets: ["node", "system","pod","container","volume"]
period: 10s
hosts: ["localhost:10255"]
The pod:
annotations:
metricbeat: "true"
system.yaml
system.yml: |-
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
processes: ['.*']
process.include_top_n:
by_cpu: 5
by_memory: 5
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
I'm commenting this bit (which works just fine removing the templating and using hint.enabled=true but it collects from everything in the cluster and it became super expensive):
# kubernetes.yml: |-
# - module: kubernetes
# metricsets:
# - node
# - system
# - pod
# - container
# - volume
# period: 10s
# host: ${NODE_NAME}
# hosts: ["localhost:10255"]
# - module: kubernetes
# enabled: true
# metricsets:
# - state_node
# - state_deployment
# - state_replicaset
# - state_statefulset
# - state_pod
# - state_container
# period: 10s
# hosts: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]
# add_metadata: true
# in_cluster: true