Hi,
I have run into a problem with the autodiscovery module on kubernetes. Filebeat is being deployed as suggested in the documentation with the addition that I wanted to have additional cloud and host metadata. Therefore I enabled the add_cloud_metadata and the add_host_metadata processor. As soon as the add_cloud_metadata processor is activated and the beat is being deployed to the cluster it crashes on some on the nodes (see log below). If the add_cloud_metadata processor is disabled everything works fine. It might have something to do with elasticsearch itself, because the beat and elasticsearch are deployed in the same cluster.
Does anybody have any suggestions regarding this error?
Greetings
Samuel
Setup information
elastic kubernetes operator: 1.0.1
elastic version: 7.6.2
filebeat version: 7.6.2
master nodes: 3
data nodes: 2
kibana nodes: 1
Filebeat configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: elasticsearch
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
processors:
- add_cloud_metadata: ~
- add_host_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
protocol: https
Debug log
Had to cut off the rest of the log because it is too large
2020-04-24T09:10:05.358Z INFO instance/beat.go:622 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-04-24T09:10:05.358Z INFO instance/beat.go:630 Beat ID: <beat-id>
2020-04-24T09:10:05.359Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-04-24T09:10:05.359Z INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"},"type": "filebeat", "uuid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}}}
2020-04-24T09:10:05.359Z INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:23:38.000Z", "version": "7.6.2"}}}
2020-04-24T09:10:05.359Z INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.8"}}}
2020-04-24T09:10:05.360Z INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-11T16:03:33Z","containerized":false,"name":"<hostname>","ip":["127.0.0.1/8","::1/128","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx"],"kernel_version":"4.14.154-128.181.amzn2.x86_64","mac":["xx:xx:xx:xx:xx:xx","xx:xx:xx:xx:xx:xx","xx:xx:xx:xx:xx:xx","xx:xx:xx:xx:xx:xx"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-04-24T09:10:05.361Z INFO add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"xxxxxxxxxxx"},"availability_zone":"eu-central-1c","image":{"id":"ami-xxxxxxx"},"instance":{"id":"i-xxxxxxxxx"},"machine":{"type":"m5.large"},"provider":"aws","region":"eu-central-1"}
2020-04-24T09:10:05.363Z INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-04-24T09:10:04.490Z"}}}
2020-04-24T09:10:05.363Z INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.2
2020-04-24T09:10:05.363Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat' as ILM is enabled.
2020-04-24T09:10:05.363Z INFO elasticsearch/client.go:174 Elasticsearch url: https://es.domain.tld:443
2020-04-24T09:10:05.363Z INFO elasticsearch/client.go:174 Elasticsearch url: https://es.domain.tld:443
2020-04-24T09:10:05.363Z INFO [publisher] pipeline/module.go:110 Beat name: <beat-name>
2020-04-24T09:10:05.364Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-04-24T09:10:05.364Z INFO instance/beat.go:439 filebeat start running.
2020-04-24T09:10:05.364Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-04-24T09:10:05.365Z INFO registrar/registrar.go:152 States Loaded from registrar: 60
2020-04-24T09:10:05.365Z INFO crawler/crawler.go:72 Loading Inputs: 0
2020-04-24T09:10:05.365Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
2020-04-24T09:10:05.365Z WARN [cfgwarn] kubernetes/kubernetes.go:62 BETA: The kubernetes autodiscover is beta
2020-04-24T09:10:05.366Z INFO kubernetes/util.go:79 kubernetes: Using node <node-name> provided in the config
2020-04-24T09:10:05.366Z INFO [autodiscover] autodiscover/autodiscover.go:104 Starting autodiscover manager
2020-04-24T09:10:05.468Z INFO log/input.go:152 Configured paths: [/var/log/containers/*<hash-1>.log]
2020-04-24T09:10:05.468Z INFO input/input.go:114 Starting input of type: container; ID: <id-1>
2020-04-24T09:10:05.469Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/<cluster-name>-es-data-1_default_elasticsearch-<hash-1>.log
2020-04-24T09:10:05.469Z INFO log/input.go:152 Configured paths: [/var/log/containers/*<hash-2>.log]
2020-04-24T09:10:05.469Z INFO input/input.go:114 Starting input of type: container; ID: <id-2>
2020-04-24T09:10:05.470Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/<cluster-name>-es-master-1_default_elasticsearch-<hash-2>.log
2020-04-24T09:10:05.471Z INFO log/input.go:152 Configured paths: [/var/log/containers/*<hash-3>.log]
2020-04-24T09:10:05.471Z INFO input/input.go:114 Starting input of type: container; ID: <id-3>
2020-04-24T09:10:05.473Z INFO log/input.go:152 Configured paths: [/var/log/containers/*<hash-4>.log]
2020-04-24T09:10:05.473Z INFO input/input.go:114 Starting input of type: container; ID: <id-4>
fatal error: concurrent map writes
fatal error: concurrent map writes
goroutine 160 [running]:
runtime.throw(0x330841d, 0x15)
/usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc000a3b7e0 sp=0xc000a3b7b0 pc=0x13e6082
runtime.mapassign_faststr(0x31f9be0, 0xc000397380, 0xc0000c08a0, 0xb, 0xc000a9e480)
/usr/local/go/src/runtime/map_faststr.go:291 +0x3fe fp=0xc000a3b848 sp=0xc000a3b7e0 pc=0x13ca19e
github.com/elastic/beats/libbeat/common.MapStr.deepUpdateMap(0xc000397380, 0xc000b112f0, 0xc000878901)
/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:90 +0x13e fp=0xc000a3b920 sp=0xc000a3b848 pc=0x18593be
github.com/elastic/beats/libbeat/common.deepUpdateValue(0x31f9be0, 0xc000397380, 0xc000b112f0, 0x1, 0xc0008789a8, 0xc0000c0888)
/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:105 +0xb0 fp=0xc000a3b958 sp=0xc000a3b920 pc=0x1859750
github.com/elastic/beats/libbeat/common.MapStr.deepUpdateMap(0xc000b10fc0, 0xc000b112c0, 0xc000397301)
/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:87 +0x25d fp=0xc000a3ba30 sp=0xc000a3b958 pc=0x18594dd
github.com/elastic/beats/libbeat/common.MapStr.DeepUpdate(...)
/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:71
github.com/elastic/beats/libbeat/processors/actions.(*addFields).Run(0xc0001fab30, 0xc000db3a00, 0xc000db3a00, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/processors/actions/add_fields.go:81 +0x51 fp=0xc000a3ba58 sp=0xc000a3ba30 pc=0x1b0b7f1
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000359f20, 0xc000db3a00, 0xc000db3a00, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 fp=0xc000a3bb08 sp=0xc000a3ba58 pc=0x1b19594
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000850600, 0xc000db3a00, 0xed6346277, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 fp=0xc000a3bbb8 sp=0xc000a3bb08 pc=0x1b19594
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc0001a3200, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:89 +0x571 fp=0xc000a3bde8 sp=0xc000a3bbb8 pc=0x1b1d851
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc0001a3200, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0xc2 fp=0xc000a3be80 sp=0xc000a3bde8 pc=0x1b1d282
github.com/elastic/beats/filebeat/channel.(*outlet).OnEvent(0xc0008507e0, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0, 0x2000100000001)
/go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x7e fp=0xc000a3bed8 sp=0xc000a3be80 pc=0x269611e
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc000521410, 0x37331e0, 0xc0008507e0)
/go/src/github.com/elastic/beats/filebeat/channel/util.go:45 +0x10d fp=0xc000a3bfc8 sp=0xc000a3bed8 pc=0x26967fd
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000a3bfd0 sp=0xc000a3bfc8 pc=0x1416ec1
created by github.com/elastic/beats/filebeat/channel.SubOutlet
/go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8