When moving from kubernetes 1.19.1 to 1.20.5 metricset:event no longer gathers .reason:FailedScheduling events.
Noticed the kubernetes event structure has changed.
1.19.1:
{
"apiVersion": "v1",
"count": 4,
"eventTime": null,
"firstTimestamp": "2021-08-05T18:05:01Z",
"involvedObject": {
"apiVersion": "v1",
"kind": "Pod",
"name": "netshoot-69c75755c-kzftr",
"namespace": "tsengineering-netshoot-dev",
"resourceVersion": "143753331",
"uid": "c5c0b600-cc55-4922-b921-8dad8a4289b3"
},
"kind": "Event",
"lastTimestamp": "2021-08-05T18:06:18Z",
"message": "0/39 nodes are available: 39 Insufficient cpu.",
"metadata": {
"creationTimestamp": "2021-08-05T18:05:01Z",
"name": "netshoot-69c75755c-kzftr.16987b2dbf489625",
"namespace": "tsengineering-netshoot-dev",
"resourceVersion": "143754124",
"selfLink": "/api/v1/namespaces/tsengineering-netshoot-dev/events/netshoot-69c75755c-kzftr.16987b2dbf489625",
"uid": "f8f0ae32-9230-4816-ae90-2e351b5bfa46"
},
"reason": "FailedScheduling",
"reportingComponent": "",
"reportingInstance": "",
"source": {
"component": "default-scheduler"
},
"type": "Warning"
}
similar event in 1.20.5 below. note a few differences:
- .metadata.selfLink is removed in 1.20
- .source is empty
- .firstTimestamp and .lastTimestamp are null
{
"action": "Scheduling",
"apiVersion": "v1",
"eventTime": "2021-08-05T16:53:44.592316Z",
"firstTimestamp": null,
"involvedObject": {
"apiVersion": "v1",
"kind": "Pod",
"name": "netshoot-5ccb8d65c7-8d5sr",
"namespace": "tsengineering-netshoot-dev",
"resourceVersion": "373641912",
"uid": "88e1db83-adfb-4e4f-8e1d-89ed98395465"
},
"kind": "Event",
"lastTimestamp": null,
"message": "0/51 nodes are available: 40 Insufficient cpu",
"metadata": {
"creationTimestamp": "2021-08-05T16:53:44Z",
"name": "netshoot-5ccb8d65c7-8d5sr.16987749e75c0e0b",
"namespace": "tsengineering-netshoot-dev",
"resourceVersion": "373807800",
"uid": "aef136c3-0e8f-4dac-abf8-24d5a1d7243d"
},
"reason": "FailedScheduling",
"reportingComponent": "default-scheduler",
"reportingInstance": "default-scheduler-ts-sharedplatform-rck-nonprod-cp-qzzjh",
"series": {
"count": 460,
"lastObservedTime": "2021-08-05T19:22:48.831892Z"
},
"source": {},
"type": "Warning"
}
Is .lastTimestamp: nullperhaps causing the event watch time comparison to fail:
Running metricbeat in debug mode with '-d "*' does not surface any errors. The events seem to be silently dropped.