Setting up Kubernetes Alerts - some k8 metrics missing

djarami726 · June 6, 2024, 2:10pm

I'm on Elastic 8.13.4, using the kubernetes integration to collect the metrics. I'm creating a series alerts equivalent to these prometheus kubernetes alerts: Awesome Prometheus alerts | Collection of alerting rules and I've had to find the equivalent metric on the elastic side. However, I have found a number of metrics that I can't find in any of the data.

Some of these metrics are:
kube_deployment_status_observed_generation,
kube_deployment_metadata_generation,
kube_statefulset_status_current_revision, kube_statefulset_status_update_revision,
kube_daemonset_status_number_misscheduled,
kube_job_spec_completions,
kube_job_status_succeeded,
kube_job_status_failed,
rest_client_requests_total

I can see references to these metrics in a elastic beats file but I don't know if this means that beats is collecting these or not.

github.com

elastic/beats/blob/main/metricbeat/module/kubernetes/_meta/test/KSM/ksm.v2.12.0.plain

# HELP kube_configmap_annotations Kubernetes annotations converted to Prometheus labels.
# TYPE kube_configmap_annotations gauge
# HELP kube_configmap_labels [STABLE] Kubernetes labels converted to Prometheus labels.
# TYPE kube_configmap_labels gauge
# HELP kube_configmap_info [STABLE] Information about configmap.
# TYPE kube_configmap_info gauge
kube_configmap_info{namespace="kube-node-lease",configmap="kube-root-ca.crt"} 1
kube_configmap_info{namespace="kube-public",configmap="cluster-info"} 1
kube_configmap_info{namespace="kube-system",configmap="coredns"} 1
kube_configmap_info{namespace="kube-system",configmap="kubeadm-config"} 1
kube_configmap_info{namespace="kube-system",configmap="kubelet-config"} 1
kube_configmap_info{namespace="default",configmap="example-redis-config"} 1
kube_configmap_info{namespace="kube-system",configmap="kube-proxy"} 1
kube_configmap_info{namespace="local-path-storage",configmap="local-path-config"} 1
kube_configmap_info{namespace="default",configmap="kube-root-ca.crt"} 1
kube_configmap_info{namespace="kube-system",configmap="extension-apiserver-authentication"} 1
kube_configmap_info{namespace="kube-system",configmap="kube-apiserver-legacy-service-account-token-tracking"} 1
kube_configmap_info{namespace="kube-system",configmap="kube-root-ca.crt"} 1
kube_configmap_info{namespace="kube-public",configmap="kube-root-ca.crt"} 1
kube_configmap_info{namespace="local-path-storage",configmap="kube-root-ca.crt"} 1

This file has been truncated. show original

I need some help on where to confirm / configure to get the metrics collected and stored in the index.

Appreciate any help.

stephenb · June 7, 2024, 5:42am

Hi @djarami726

Assuming you are running KSM... I think the _ are replaced with dots... I did not find all but I see most. And of course they will not show up if you are not generating them

Example

djarami726 · June 17, 2024, 6:03pm

Thank you for response. I was able to find the two that you mentioned above, but there are others that don't seem to be coming through the pipeline and I've been able that they are accessable from the kubernetes side.

Yes, I did look for them with the . instead of the _ and all kinds of similarity searching too.

Here is a list of the metrics I wasn't able to find:
kube_deployment_status_observed_generation
kube_deployment_metadata_generation
kube_statefulset_status_current_revision
kube_statefulset_status_update_revision
kube_daemonset_status_number_misscheduled
kube_job_spec_completions
kube_job_status_succeeded
kube_job_status_failed
rest_client_requests_total
apiserver_client_certificate_expiration_seconds_count

Thanks for your help

Topic		Replies	Views
Prometheus Federation Metricbeat - No index and document in ES Beats metricbeat	1	515	May 18, 2021
[QUESTION/HELP] Metricbeat kubernetes state_node labels Beats metricbeat	2	342	November 27, 2020
Watcher repository by metricbeat module (kubernetes, system) Beats metricbeat	4	371	November 5, 2019
Metricbeat on kubernetes, conflicting configurations Beats metricbeat	3	435	November 15, 2021
Kubernetes.node.status.* metrics not available for alerting Beats metricbeat	3	406	February 22, 2023

Setting up Kubernetes Alerts - some k8 metrics missing

Related topics