Hello,
We currently have a Kubernetes cluster that has a Prometheus Installation.
I'm looking to export the Prometheus gathered metrics to Elastic so we can create Kibana dashboard; using the Federation approach (so we do not have to change all annotations).
Here's the Kubernetes YAML.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-config
namespace: monitoring
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
logging.level: info
metricbeat.config.modules:
# Mounted `metricbeat-daemonset-modules` configmap:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: monitoring
labels:
k8s-app: metricbeat
data:
kubernetes.yml: |-
- module: prometheus
enabled: true
period: 180s
metricsets: ["collector"]
hosts: ["prometheus-kube-prometheus-prometheus:9090"]
metrics_path: '/federate'
headers:
accept: "text/plain"
query:
'match[]': '{__name__!=""}'
timeout: 20s
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: metricbeat
namespace: monitoring
labels:
k8s-app: metricbeat
spec:
replicas: 1
selector:
matchLabels:
k8s-app: metricbeat
template:
metadata:
labels:
k8s-app: metricbeat
spec:
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
imagePullPolicy: Always
image: docker.elastic.co/beats/metricbeat:7.12.0
args: [
"-c", "/etc/metricbeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: [redacted]
- name: ELASTICSEARCH_PORT
value: "9243"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: [redacted]
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: metricbeat-deployment-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-deployment-modules
---
My problem is that metricbeat does not "seems" to send data to Elastic.
On Elastic, there's no metricbeat-* indices.
Here's the log of the metricbeat pod on Kubernetes
2021-04-20T14:01:27.220Z INFO instance/beat.go:660 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2021-04-20T14:01:27.229Z INFO instance/beat.go:668 Beat ID: 975e3069-3277-4025-af26-1bf0231accf1
2021-04-20T14:01:27.236Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-04-20T14:01:27.236Z INFO [beat] instance/beat.go:996 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "975e3069-3277-4025-af26-1bf0231accf1"}}}
2021-04-20T14:01:27.237Z INFO [beat] instance/beat.go:1005 Build info {"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:28:33.000Z", "version": "7.12.0"}}}
2021-04-20T14:01:27.237Z INFO [beat] instance/beat.go:1008 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.15.8"}}}
2021-04-20T14:01:27.239Z INFO [beat] instance/beat.go:1012 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-04-10T22:30:54Z","containerized":true,"name":"metricbeat-69f9758c46-dfmhj","ip":["127.0.0.1/8","10.4.1.115/24"],"kernel_version":"5.4.89+","mac":["d2:4a:61:eb:7f:f8"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"44ccc339d95bd51386fcfc5d8f041927"}}}
2021-04-20T14:01:27.239Z INFO [beat] instance/beat.go:1041 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 6, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-04-20T14:01:26.820Z"}}}
2021-04-20T14:01:27.240Z INFO instance/beat.go:304 Setup Beat: metricbeat; Version: 7.12.0
2021-04-20T14:01:27.240Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-7.12.0' as ILM is enabled.
2021-04-20T14:01:27.241Z INFO eslegclient/connection.go:99 elasticsearch url: https://[redacted].gcp.elastic-cloud.com:9243
2021-04-20T14:01:27.243Z INFO [publisher] pipeline/module.go:113 Beat name: metricbeat-69f9758c46-dfmhj
2021-04-20T14:01:27.247Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:105 add_cloud_metadata: hosting provider type detected as gcp, [redacted]
2021-04-20T14:01:27.269Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-04-20T14:01:27.269Z INFO instance/beat.go:468 metricbeat start running.
2021-04-20T14:01:27.271Z INFO cfgfile/reload.go:164 Config reloader started
2021-04-20T14:01:27.272Z INFO cfgfile/reload.go:224 Loading of config files completed.
2021-04-20T14:01:37.353Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://[redacted].gcp.elastic-cloud.com:9243))
2021-04-20T14:01:37.354Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-04-20T14:01:37.354Z INFO [publisher] pipeline/retry.go:223 done
2021-04-20T14:01:37.492Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2021-04-20T14:01:37.496Z INFO [license] licenser/es_callback.go:51 Elasticsearch license: Platinum
2021-04-20T14:01:37.498Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2021-04-20T14:01:37.501Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-04-20T14:01:37.504Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2021-04-20T14:01:37.504Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-04-20T14:01:37.504Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-7.12.0 {now/d}-000001}' as ILM is enabled.
2021-04-20T14:01:37.504Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-7.12.0-*' as ILM is enabled.
2021-04-20T14:01:37.504Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.12.0 {now/d}-000001} as ILM is enabled.
2021-04-20T14:01:37.504Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-20T14:01:37.536Z INFO template/load.go:97 Template metricbeat-7.12.0 already exists and will not be overwritten.
2021-04-20T14:01:37.537Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2021-04-20T14:01:37.542Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
2021-04-20T14:01:37.544Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://[redacted].gcp.elastic-cloud.com:9243)) established
I tried letting the pod runs for 1 hour, and nothing never gets to the Elastic Cluster.
I tried to curl the Prometheus from the metricbeat pod, and it works as intended.
I also tried to connect to the cluster using curl, and it works.