[Filebeat] Panic in K8s autodiscover

Elastic Stack Beats
1

Hi,

after upgrading from 7.5.2 to 7.9.2 I observe sporadic container crashes with the following stacktrace:

fatal error: concurrent map read and map write

goroutine 5766 [running]:
runtime.throw(0x3d8cd28, 0x21)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0009c9350 sp=0xc0009c9320 pc=0x16577d2
runtime.mapaccess2_faststr(0x38d5520, 0xc0016ef470, 0xc0016f6380, 0xe, 0xc00249cb40, 0x0)
	/usr/local/go/src/runtime/map_faststr.go:116 +0x47c fp=0xc0009c93c0 sp=0xc0009c9350 pc=0x163616c
github.com/elastic/beats/v7/libbeat/common/kubernetes/k8skeystore.(*KubernetesKeystoresRegistry).GetKeystore(0xc003adbc20, 0xc005fbade0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/common/kubernetes/k8skeystore/kubernetes_keystore.go:79 +0x110 fp=0xc0009c9498 sp=0xc0009c93c0 pc=0x2b15240
github.com/elastic/beats/v7/libbeat/autodiscover.Builders.GetConfig(0x0, 0x0, 0x0, 0x4255120, 0xc003adbc20, 0xc005fbade0, 0xc003adbc20, 0xc006850c30, 0x0)
	/go/src/github.com/elastic/beats/libbeat/autodiscover/builder.go:102 +0x281 fp=0xc0009c9550 sp=0xc0009c9498 pc=0x1e7e0a1
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish(0xc000d44e70, 0xc006850c30)
	/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/kubernetes.go:148 +0x236 fp=0xc0009c9620 sp=0xc0009c9550 pc=0x2b17436
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish-fm(0xc006850c30)
	/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/kubernetes.go:141 +0x34 fp=0xc0009c9640 sp=0xc0009c9620 pc=0x2b1fd84
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emitEvents(0xc001967c00, 0xc0001c5000, 0x3d392f8, 0x4, 0xc0016b02c0, 0x1, 0x1, 0xc00166a380, 0x1, 0x1)
	/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:428 +0xa82 fp=0xc0009c9f50 sp=0xc0009c9640 pc=0x2b1be92
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit(0xc001967c00, 0xc0001c5000, 0x3d392f8, 0x4)
	/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:270 +0x95 fp=0xc0009c9fb0 sp=0xc0009c9f50 pc=0x2b1b385
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate.func1()
	/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:142 +0x48 fp=0xc0009c9fe0 sp=0xc0009c9fb0 pc=0x2b1fb58
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0009c9fe8 sp=0xc0009c9fe0 pc=0x168b5a1
created by time.goFunc
	/usr/local/go/src/time/sleep.go:168 +0x44

I use Kubernetes autodiscover with a lot of conditions, e.g.

  filebeat.autodiscover:
    providers:
      - type: kubernetes
        hints.enabled: false
        templates:
          - condition:
              or:
                - equals:
                    kubernetes.container.name: x
                - equals:
                    kubernetes.container.name: y
                - equals:
                    kubernetes.container.name: z
            config:
              - type: container
                ignore_older: 10m
                multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s'
                multiline.negate: true
                multiline.match: after
                paths:
                  - /var/lib/docker/containers/${data.kubernetes.container.id}/*-json.log
                  - /var/lib/docker/containers/${data.kubernetes.container.id}/*-json.log.*
                fields:
                  cloud_region: "xyz"
                  event:
                    category: "xyz"
                fields_under_root: true
                tail_files: true
                clean_removed: true
                scan_frequency: "5s"
[...]

Looks like a bug to me and I could not find a related issue. Should I open one?

Regards
Bernd

2

Hi Bernd! Definitely, please open an issue. It would be also beneficial for us to write more details, how often does it appear, is it easy to reproduce (and how). Thank you in advance!

3

Opened https://github.com/elastic/beats/issues/21843

4

Thanks for reporting this @boernd and @mtojek for the escalation. PR with the fix on its way.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.