I have deployed Filebeat -> Logstash -> elasticseach -> kibana in OKD-3.11 running on Openstack Centos-7.6 VMs. Using oss docker images for the deployment.
docker.elastic.co/beats/filebeat-oss:7.9.3
docker.elastic.co/logstash/logstash-oss:7.9.3
docker.elastic.co/elasticsearch/elasticsearch-oss:7.9.3
docker.elastic.co/kibana/kibana-oss:7.9.3
The filebeat pods are restarting with "fatal error: concurrent map read and map write".
And I don't see any issues from the filebeat functionality or performance but just want to know if this is expected.
Some of the observations here are -
The filebeat pod restarts after every time master api & etcd containers restarts.
Filebeat configuration file -
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
tags:
- "kube-logs"
templates:
- condition:
or:
- contains:
kubernetes.pod.name: "ne-mgmt"
- contains:
kubernetes.pod.name: "list-manager"
- contains:
kubernetes.pod.name: "scheduler-mgmt"
- contains:
kubernetes.pod.name: "sync-ne"
- contains:
kubernetes.pod.name: "file-manager"
- contains:
kubernetes.pod.name: "dash-board"
- contains:
kubernetes.pod.name: "ne-db-manager"
- contains:
kubernetes.pod.name: "config-manager"
- contains:
kubernetes.pod.name: "report-manager"
- contains:
kubernetes.pod.name: "clean-backup"
- contains:
kubernetes.pod.name: "warrior"
- contains:
kubernetes.pod.name: "ne-backup"
- contains:
kubernetes.pod.name: "ne-restore"
config:
- type: container
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
logging.level: debug
processors:
- drop_event:
when.or:
- equals:
kubernetes.namespace: "kube-system"
- equals:
kubernetes.namespace: "default"
- equals:
kubernetes.namespace: "logging"
output.logstash:
hosts: ["logstash-service.logging:5044"]
index: filebeat
pretty: true
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
The error from the logs of filebeat -
fatal error: concurrent map read and map write
goroutine 455764 [running]:
runtime.throw(0x29bcfcc, 0x21)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc01836d330 sp=0xc01836d300 pc=0xedc6b2
runtime.mapaccess2_faststr(0x26e44e0, 0xc0017c1920, 0xc00d4ea170, 0xe, 0xc00e351e90, 0x0)
/usr/local/go/src/runtime/map_faststr.go:116 +0x47c fp=0xc01836d3a0 sp=0xc01836d330 pc=0xebb04c
github.com/elastic/beats/v7/libbeat/common/kubernetes/k8skeystore.(*KubernetesKeystoresRegistry).GetKeystore(0xc001e53680, 0xc01205f980, 0x28a5aa0, 0xc01836d4b0)
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/k8skeystore/kubernetes_keystore.go:79 +0x110 fp=0xc01836d478 sp=0xc01836d3a0 pc=0x2375330
github.com/elastic/beats/v7/libbeat/autodiscover/template.Mapper.GetConfig(0xc000010ed8, 0x1, 0x1, 0x2cf31e0, 0xc00004ed80, 0x2caf8e0, 0xc001e53680, 0xc01205f980, 0xc000541500, 0x7f68ad5767d0, ...)
/go/src/github.com/elastic/beats/libbeat/autodiscover/template/config.go:95 +0x413 fp=0xc01836d550 sp=0xc01836d478 pc=0x17009e3
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish(0xc000d0c2c0, 0xc01205f980)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/kubernetes.go:143 +0xae fp=0xc01836d620 sp=0xc01836d550 pc=0x237739e
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish-fm(0xc01205f980)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/kubernetes.go:141 +0x34 fp=0xc01836d640 sp=0xc01836d620 pc=0x237fe74
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emitEvents(0xc00060bf80, 0xc014ad0400, 0x29855ef, 0x4, 0xc006642160, 0x1, 0x1, 0xc000f44580, 0x1, 0x1)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:428 +0xa82 fp=0xc01836df50 sp=0xc01836d640 pc=0x237bf82
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit(0xc00060bf80, 0xc014ad0400, 0x29855ef, 0x4)
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:270 +0x95 fp=0xc01836dfb0 sp=0xc01836df50 pc=0x237b475
github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate.func1()
/go/src/github.com/elastic/beats/libbeat/autodiscover/providers/kubernetes/pod.go:142 +0x48 fp=0xc01836dfe0 sp=0xc01836dfb0 pc=0x237fc48
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc01836dfe8 sp=0xc01836dfe0 pc=0xf102c1
created by time.goFunc
/usr/local/go/src/time/sleep.go:168 +0x44
The log actually contains around 20 lakhs of lines so just pasted a part of it.
Partial log - The Go Playground