Hi,
after upgrading from 7.5.2 to 7.9.2 I observe sporadic container crashes with the following stacktrace:
fatal error: concurrent map read and map write
goroutine 5766 [running]:
runtime.throw(0x3d8cd28, 0x21)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0009c9350 sp=0xc0009c9320 pc=0x16577d2
runtime.mapaccess2_faststr(0x38d5520, 0xc0016ef470, 0xc0016f6380, 0xe, 0xc00249cb40, 0x0)
/usr/local/go/src/runtime/map_faststr.go:116 +0x47c fp=0xc0009c93c0 sp=0xc0009c9350 pc=0x163616c
github.com/elastic/beats/v7/libbeat/common/kubernetes/k8skeystore.(*KubernetesKeystoresRegistry).GetKeystore(0xc003adbc20, 0xc005fbade0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/k8skeystore/kubernetes_keystore.go:79 +0x110 fp=0xc0009c9498 sp=0xc0009c93c0 pc=0x2b15240
github.com/elastic/beats/v7/libbeat/autodiscover.Builders.GetConfig(0x0, 0x0, 0x0, 0x4255120, 0xc003adbc20, 0xc005fbade0, 0xc003adbc20, 0xc006850c30, 0x0)
/go/src/github.com/elastic/beats/libbeat/autodiscover/builder.go:102 +0x281 fp=0xc0009c9550 sp=0xc0009c9498 pc=0x1e7e0a1
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish(0xc000d44e70, 0xc006850c30)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/kubernetes.go:148 +0x236 fp=0xc0009c9620 sp=0xc0009c9550 pc=0x2b17436
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish-fm(0xc006850c30)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/kubernetes.go:141 +0x34 fp=0xc0009c9640 sp=0xc0009c9620 pc=0x2b1fd84
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emitEvents(0xc001967c00, 0xc0001c5000, 0x3d392f8, 0x4, 0xc0016b02c0, 0x1, 0x1, 0xc00166a380, 0x1, 0x1)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:428 +0xa82 fp=0xc0009c9f50 sp=0xc0009c9640 pc=0x2b1be92
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit(0xc001967c00, 0xc0001c5000, 0x3d392f8, 0x4)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:270 +0x95 fp=0xc0009c9fb0 sp=0xc0009c9f50 pc=0x2b1b385
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate.func1()
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:142 +0x48 fp=0xc0009c9fe0 sp=0xc0009c9fb0 pc=0x2b1fb58
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0009c9fe8 sp=0xc0009c9fe0 pc=0x168b5a1
created by time.goFunc
/usr/local/go/src/time/sleep.go:168 +0x44
I use Kubernetes autodiscover with a lot of conditions, e.g.
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: false
templates:
- condition:
or:
- equals:
kubernetes.container.name: x
- equals:
kubernetes.container.name: y
- equals:
kubernetes.container.name: z
config:
- type: container
ignore_older: 10m
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\s'
multiline.negate: true
multiline.match: after
paths:
- /var/lib/docker/containers/${data.kubernetes.container.id}/*-json.log
- /var/lib/docker/containers/${data.kubernetes.container.id}/*-json.log.*
fields:
cloud_region: "xyz"
event:
category: "xyz"
fields_under_root: true
tail_files: true
clean_removed: true
scan_frequency: "5s"
[...]
Looks like a bug to me and I could not find a related issue. Should I open one?
Regards
Bernd