Hi all,
I'm running an ECK on a Kubernetes cluster and am using FileBeat and MetricBeat to send monitoring information to Elastic.
For the most part, things are running smoothly, except for the MetricBeat Deployment, which is not providing any output data (the DaemonSet works fine).
I've confirmed that the KubeStateMetrics pod is up and running and gathering data as expected, so I assume that either MetricBeat has trouble connecting to KubeStateMetrics or to Elastic.
The logs for the MetricBeat pod shows no connection errors however.
In hopes of isolating the issue, I've changed the 'Host' properties of the Metricbeat deployment for both the Elastic and Kube-State-Metrics services to non-existent endpoints and changed the auth credentials, to see if that would throw any useful errors.
I have tried (in separate steps):
- Invalid host for Kube-State-Metrics
- Invalid host for Elastic
- Invalid username/password for Elastic
- Invalid certificate for Elastic
In all cases, no errors are shown at all. Activating 'debug' logging also yields no useful information.
The Metricbeat logs simply shows the 'Non-zero metrics collected' INFO messages.
Seemingly it does not matter whether I provide valid or invalid hosts or credentials.
This strikes me as unexpected behavior, and makes me think I'm overlooking something.
Can someone verify that under normal circumstances, connection errors should be shown?
And are there further steps I can take to debug/isolate the problem?
Deployment config:
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-config
namespace: elastic
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
- add_kubernetes_metadata:
in_cluster: true
setup.ilm.enabled: false
output.elasticsearch:
hosts: ['https://elastic-es-http:9200']
ssl.certificate_authorities: ["/usr/share/elastic/certs/ca.crt"]
ssl.certificate: '/usr/share/elastic/certs/tls.crt'
ssl.key: '/usr/share/elastic/certs/tls.key'
username: '{username}'
password: "{password}"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: elastic
labels:
k8s-app: metricbeat
data:
# This module requires `kube-state-metrics` up and running under `kube-system` namespace
kubernetes.yml: |-
- module: kubernetes
labels.dedot: true
annotations.dedot: true
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
- state_statefulset
# Uncomment this to get k8s events:
- event
period: 10s
hosts: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]
add_metadata: true
in_cluster: true
enabled: true
---
# Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: metricbeat
namespace: elastic
labels:
k8s-app: metricbeat
spec:
template:
metadata:
creationTimestamp: ~
labels:
k8s-app: metricbeat
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: agentpool
operator: In
values:
- elastic
containers:
- args:
- "-c"
- /etc/metricbeat.yml
- "-e"
image: "docker.elastic.co/beats/metricbeat-oss:7.5.0"
imagePullPolicy: IfNotPresent
name: metricbeat
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/elastic/certs/
name: elastic-internal-http-certificates
readOnly: true
- mountPath: /etc/metricbeat.yml
name: config
readOnly: true
subPath: metricbeat.yml
- mountPath: /usr/share/metricbeat/modules.d
name: modules
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: metricbeat
serviceAccountName: metricbeat
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: restriction
operator: Equal
value: elastic
volumes:
- configMap:
defaultMode: 384
name: metricbeat-deployment-config
name: config
- name: elastic-internal-http-certificates
secret:
defaultMode: 420
optional: false
secretName: elastic-es-http-certs-internal
- configMap:
defaultMode: 384
name: metricbeat-deployment-modules
name: modules