Module: kubernetes, metricset: container - is missing exported fields from docs (such as status?)

Hey there,
I'm trying to get the status of a pod to alert for Crashloop Backoffs and other non Ready states.

To achieve that I've tried the docs:

Both say that should be a field "kubernetes.{pod/container}.status.phase" but it doesn't exist.

An example of a raw document ("pod" is the same) shows that indeed this data is not being sent, is shown below.

Metricbeat version: 7.9.0
The output is directly to elasticsearch (version 7.9.0)
System: GKE (Google Kubernetes Engine)
Kubernetes Master version: v1.15.12-gke.2
Kubernetes node version: v1.15.12-gke.2 (the same)

What am I missing here?

Thanks!
Ilai

Document example without "...status.phase":

{
  "_index": "metricbeat-devops-7.9.0-2020.09.13-000002",
  "_type": "_doc",
  "_id": "...",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-09-13T18:55:43.012Z",
    "event": {
      "dataset": "kubernetes.container",
      "module": "kubernetes",
      "duration": 29417277
    },
    "metricset": {
      "name": "container",
      "period": 15000
    },
    "service": {
      "address": "localhost:10255",
      "type": "kubernetes"
    },
    "kubernetes": {
      "namespace": "tip",
      "node": {
        "name": "..."
      },
      "pod": {
        "name": "..."
      },
      "container": {
        "rootfs": {
          "available": {
            "bytes": 93795864576
          },
          "capacity": {
            "bytes": 101241290752
          },
          "used": {
            "bytes": 1089536
          },
          "inodes": {
            "used": 16
          }
        },
        "logs": {
          "capacity": {
            "bytes": 101241290752
          },
          "used": {
            "bytes": 32768
          },
          "inodes": {
            "used": 202197,
            "free": 6056523,
            "count": 6258720
          },
          "available": {
            "bytes": 93795864576
          }
        },
        "start_time": "2020-09-10T20:49:58Z",
        "name": "...",
        "cpu": {
          "usage": {
            "nanocores": 3281449,
            "core": {
              "ns": 1033744763620
            },
            "node": {
              "pct": 0.00082036225
            },
            "limit": {
              "pct": 0.00082036225
            }
          }
        },
        "memory": {
          "workingset": {
            "bytes": 361504768
          },
          "rss": {
            "bytes": 357220352
          },
          "pagefaults": 173250,
          "majorpagefaults": 0,
          "available": {
            "bytes": 0
          },
          "usage": {
            "bytes": 362520576,
            "node": {
              "pct": 0.022992100865353617
            },
            "limit": {
              "pct": 0.022992100865353617
            }
          }
        }
      }
    },
    "host": {
      "name": "..."
    },
    "agent": {
      "name": "...",
      "type": "metricbeat",
      "version": "7.9.0",
      "hostname": "...",
      "ephemeral_id": "...",
      "id": "..."
    },
    "ecs": {
      "version": "1.5.0"
    },
    "cloud": {
      "provider": "gcp",
      "instance": {
        "name": "...",
        "id": "1126470843320709070"
      },
      "machine": {
        "type": "n1-standard-4"
      },
      "availability_zone": "us-central1-b",
      "project": {
        "id": "..."
      }
    }
  },
  "fields": {
    "kubernetes.container.start_time": [
      "2020-09-10T20:49:58.000Z"
    ],
    "@timestamp": [
      "2020-09-13T18:55:43.012Z"
    ]
  },
  "sort": [
    1600023343012
  ]
}

Hi @Ilai_Fallach, welcome to the Elastic community forums!

These fields are produced by the state_container and state_pod metricsets. Do you have these metricsets enabled in your kubernetes module configuration file? Note that for these metricsets to be able to collect the necessary data from Kubernetes you will need to run kube-state-metrics service.

Shaunak

1 Like

Thank you @shaunak,

I've installed the kube-state-metrics and everything is working now.

Thanks for the help,
Ilai