I deployed the Prometheus、apm-server、elastic、kibana 、metricbeat in k8s. I want to show apm metrics from prometheus data. what should I do?
Kibana version: 7.13.0
Elasticsearch version: 7.13.0
APM Server version: 7.13.0
Steps to reproduce:
- run es,kibana,apm
- The apm-server config
apm-server.yml: |-
apm-server:
host: "0.0.0.0:8200"
queue.mem.events: 2560
output.elasticsearch:
hosts: ['http://elasticsearch.elastic-system.svc:80']
workers: 1
bulk_max_size: 2560
setup.kibana:
host: "http://kibana-svc.elastic-system.svc:80"
processors:
- drop_fields:
fields: ["client.ip"]
ignore_missing: true
- run metricbeat,the configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-config
namespace: elastic-system
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
# Mounted `metricbeat-daemonset-modules` configmap:
path: /usr/share/metricbeat/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: true
metricbeat.autodiscover:
providers:
- type: kubernetes
scope: cluster
node: ${NODE_NAME}
unique: true
templates:
- config:
- module: kubernetes
hosts: ["kube-state-metrics:8080"]
period: 10s
add_metadata: true
metricsets:
- state_node
- state_deployment
- state_daemonset
- state_replicaset
- state_pod
- state_container
- state_cronjob
- state_resourcequota
- state_statefulset
- state_service
- module: kubernetes
metricsets:
- apiserver
hosts: ["https://xxxxxxx"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
period: 30s
- type: kubernetes
include_annotations: ["*"]
templates:
- condition:
contains:
kubernetes.annotations.prometheus.io/scrape: "true"
config:
- module: prometheus
metricsets: ["collector"]
hosts: "prometheus-svc.ops.svc:80"
# - module: prometheus
# hosts: ["prometheus-svc.ops.svc:80"]
# metrics_path: /api/v1/label/__name__/values
# Uncomment this to get k8s events:
#- module: kubernetes
# metricsets:
# - event
# To enable hints based autodiscover uncomment this:
#- type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
processors:
- add_cloud_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-modules
namespace: elastic-system
labels:
k8s-app: metricbeat
data:
system.yml: |-
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
#- core
#- diskio
#- socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
kubernetes.yml: |-
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
host: ${NODE_NAME}
hosts: ["https://${NODE_NAME}:10250"]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.verification_mode: "none"
-
Add metricbeat Indices
-
I can show the metrics on kibana,like this:
-
apm metrics can't show the metrics.
How to fix it?