Nil pointer dereference exception in libbeat/processors/add_kubernetes_metadata/kubernetes.go:161

okke-formsma · March 12, 2018, 8:31am

I am seeing an issue in Filebeat 6.2.2. I'm running libbeat in a container on Kubernetes. Every other hour, the process will crash with the following stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
	March 11th 2018, 11:51:44.107	167,691	[signal SIGSEGV: segmentation violation code=0x1 addr=0x158 pc=0x13dd8ba]

goroutine 76 [running]:
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*ContainerIndexer).GetIndexes(0xc42040a1f0, 0x0, 0xc4203ec900, 0xc4204bd620, 0xc4204bd560)
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/indexers.go:161 +0x3a
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*Indexers).GetIndexes(0xc4203ec900, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/indexers.go:76 +0x230
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*kubernetesAnnotator).removePod(0xc42018e620, 0x0)
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:188 +0x3d
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*kubernetesAnnotator).(github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.removePod)-fm(0x0)
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:159 +0x34
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.processEvent(0xc4205bded0, 0xc420956060)
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:175 +0x78
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*kubernetesAnnotator).worker(0xc42018e620)
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:159 +0x26f
created by github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.newKubernetesAnnotator
/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:113 +0x9f7

The offending line of code is the following:

github.com

elastic/beats/blob/8719ad48ceaaa409d27547bd190bc165e5108066/libbeat/processors/add_kubernetes_metadata/kubernetes.go#L161


// worker watches pod events and keeps a map of metadata
func (k *kubernetesAnnotator) worker() {
	for {
		select {
		case event := <-k.startListener.Events():
			processEvent(k.addPod, event)


		case event := <-k.stopListener.Events():
			processEvent(k.removePod, event)


		case event := <-k.updateListener.Events():
			processEvent(k.removePod, event)
			processEvent(k.addPod, event)
		}
	}
}


// Run pod actions while handling errors
func processEvent(f func(pod *kubernetes.Pod), event bus.Event) {
	pod, ok := event["pod"].(*kubernetes.Pod)
	if !ok {

Any clue to why this might happen?

exekias · March 12, 2018, 12:19pm

Hi @okke-formsma,

Thank you for reporting this! Actually this was a known issue that was recently fixed: https://github.com/elastic/beats/issues/6372

Next version (6.2.3) will be released soon, and it will pack a fix for this!

Best regards

okke-formsma · March 12, 2018, 12:47pm

Hi @exekias,

Thanks for the reply! I hadn't seen that github issue.

Regards

Okke

system · April 9, 2018, 12:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.