Hi,
I'm running Filebeat 7.8.1 on ECK, however, it seems that the harvester is not starting. I'm not getting any logs in Elasticsearch.
After some investigation, I found that the /usr/share/filebeat/data/registry/filebeat/data.json file is empty, and I suspect this is the reason it is not working. I'm not seeing any error logs on the pod.
Anyone with a similar issue or any advice?
Filebeat config:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
namespace: elastic-system
spec:
type: filebeat
version: 7.8.1
elasticsearchRef:
name: elasticsearch
kibanaRef:
name: kibana
config:
filebeat:
autodiscover:
providers:
- type: kubernetes
node: ${HOSTNAME}
hints:
enabled: true
default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
logging.level: debug
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
tolerations:
- key: "nodetype"
operator: "Equal"
value: "elk"
effect: "NoSchedule"
containers:
- name: filebeat
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- events
- nodes
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: elastic-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
Pod Logs:
2020-08-13T09:15:10.928Z INFO instance/beat.go:647 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-08-13T09:15:10.929Z INFO instance/beat.go:655 Beat ID: 155adddd-2cba-4a5f-bff5-ff7e22496e0d
2020-08-13T09:15:10.930Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"xxxxxxxxxxx"},"availability_zone":"eu-west-1a","image":{"id":"ami-0de1b659c5a3fd30a"},"instance":{"id":"i-xxxxxxxxxxxxx"},"machine":{"type":"m5.4xlarge"},"provider":"aws","region":"eu-west-1"}
2020-08-13T09:15:11.227Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-08-13T09:15:11.227Z INFO [beat] instance/beat.go:983 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "155adddd-2cba-4a5f-bff5-ff7e22496e0d"}}}
2020-08-13T09:15:11.227Z INFO [beat] instance/beat.go:992 Build info {"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T15:12:45.000Z", "version": "7.8.1"}}}
2020-08-13T09:15:11.227Z INFO [beat] instance/beat.go:995 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.13.10"}}}
2020-08-13T09:15:11.329Z INFO [beat] instance/beat.go:999 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-06-25T11:16:28Z","containerized":false,"name":"ip-172-26-25-220.clickatell.com","ip":["127.0.0.1/8","::1/128","172.26.25.220/23","fe80::cb:abff:fedd:4a54/64","172.26.25.247/23","fe80::7d:58ff:fec8:5410/64","fe80::d0f3:2bff:fee9:e623/64","fe80::80e7:4aff:fe57:f7f9/64","fe80::f842:d7ff:fe14:aa7f/64","fe80::cc9f:2dff:fe82:20cd/64","fe80::e813:47ff:fe98:6415/64","fe80::dc5a:eff:feb2:2aed/64","fe80::38bc:5cff:fedc:fe64/64","fe80::d0c4:38ff:feed:f536/64","fe80::f0d2:14ff:fec8:a8cc/64","fe80::84:5bff:feed:f93c/64","fe80::102e:b5ff:fe0a:781c/64","172.26.24.220/23","fe80::60:51ff:fe71:3e40/64","fe80::10f1:4fff:fe53:60bd/64","fe80::9484:c0ff:fe67:4edf/64","fe80::2cbd:a0ff:fe6c:7d22/64","fe80::d8a2:a6ff:fe6a:ae1d/64","fe80::ece9:d1ff:fe4b:b4f1/64","fe80::50d4:c3ff:feae:ba94/64","fe80::d85d:1ff:fe7e:7798/64","fe80::38c0:b9ff:fe25:fe60/64","fe80::807b:45ff:fef5:a4ad/64","fe80::6cf2:b9ff:fef7:a61c/64","fe80::903a:b9ff:feac:7e77/64","fe80::34dc:68ff:fedd:642b/64","fe80::30f2:a4ff:feae:97e2/64","fe80::74f9:fdff:fea0:838d/64","fe80::a0c0:eaff:fee1:576/64","fe80::f058:35ff:fe4e:d5fe/64","fe80::b06d:a7ff:fee8:bd65/64","fe80::6855:89ff:fe88:70c9/64","fe80::ca7:c7ff:fef6:14e7/64","fe80::38ac:73ff:fe29:3f40/64","fe80::282d:8aff:fee1:59a1/64","fe80::8c4:9aff:fe2b:5406/64","fe80::9cea:99ff:fe7e:2c1b/64","fe80::4008:edff:fe10:fbd0/64","fe80::7c71:6eff:fe9e:4453/64","fe80::70fa:2eff:fecb:c74c/64","fe80::ccfb:caff:fe15:6f0f/64","fe80::b8f3:dbff:fe84:9fd9/64","fe80::5806:48ff:fe86:ec9e/64","fe80::2cc2:9eff:fef5:a96c/64","fe80::1c93:e2ff:fe40:aaa5/64","fe80::bcf7:adff:fe57:1d51/64","172.26.24.200/23","fe80::d1:99ff:fe90:3532/64","fe80::74af:7eff:fe44:3669/64","fe80::10cf:beff:fe3c:c39e/64","fe80::fc6d:a3ff:fe08:b833/64","fe80::c3e:dfff:fe3c:a3b1/64","fe80::8459:76ff:fe75:ffcb/64","fe80::d840:5dff:febc:6aef/64","fe80::8894:1ff:fe13:bad2/64","fe80::7cb9:97ff:fefb:273b/64","fe80::878:5cff:fe49:349a/64","fe80::c452:4aff:fef9:862c/64","fe80::14da:17ff:fe5e:3ab7/64","fe80::fc4a:3dff:fea8:9ebb/64","fe80::c52:34ff:fe4e:eb8d/64","fe80::248d:9dff:fefe:51cd/64","fe80::1804:2ff:fe61:bae/64","fe80::f8ad:aaff:fe16:3275/64","fe80::6c2d:27ff:feb3:27df/64","fe80::e06f:ddff:fe81:12ab/64","fe80::24d3:42ff:fec1:d63a/64","fe80::6cdd:14ff:fede:e1e8/64","fe80::4ce3:7cff:feb6:ae30/64","fe80::b439:54ff:feb7:9b1e/64","fe80::e455:90ff:fe80:54b9/64","fe80::502e:1cff:fe1b:1f41/64"],"kernel_version":"4.14.181-140.257.amzn2.x86_64","mac":["02:cb:ab:dd:4a:54","02:7d:58:c8:54:10","d2:f3:2b:e9:e6:23","82:e7:4a:57:f7:f9","fa:42:d7:14:aa:7f","ce:9f:2d:82:20:cd","ea:13:47:98:64:15","de:5a:0e:b2:2a:ed","3a:bc:5c:dc:fe:64","d2:c4:38:ed:f5:36","f2:d2:14:c8:a8:cc","02:84:5b:ed:f9:3c","12:2e:b5:0a:78:1c","02:60:51:71:3e:40","12:f1:4f:53:60:bd","96:84:c0:67:4e:df","2e:bd:a0:6c:7d:22","da:a2:a6:6a:ae:1d","ee:e9:d1:4b:b4:f1","52:d4:c3:ae:ba:94","da:5d:01:7e:77:98","3a:c0:b9:25:fe:60","82:7b:45:f5:a4:ad","6e:f2:b9:f7:a6:1c","92:3a:b9:ac:7e:77","36:dc:68:dd:64:2b","32:f2:a4:ae:97:e2","76:f9:fd:a0:83:8d","a2:c0:ea:e1:05:76","f2:58:35:4e:d5:fe","b2:6d:a7:e8:bd:65","6a:55:89:88:70:c9","0e:a7:c7:f6:14:e7","3a:ac:73:29:3f:40","2a:2d:8a:e1:59:a1","0a:c4:9a:2b:54:06","9e:ea:99:7e:2c:1b","42:08:ed:10:fb:d0","7e:71:6e:9e:44:53","72:fa:2e:cb:c7:4c","ce:fb:ca:15:6f:0f","ba:f3:db:84:9f:d9","5a:06:48:86:ec:9e","2e:c2:9e:f5:a9:6c","1e:93:e2:40:aa:a5","be:f7:ad:57:1d:51","02:d1:99:90:35:32","76:af:7e:44:36:69","12:cf:be:3c:c3:9e","fe:6d:a3:08:b8:33","0e:3e:df:3c:a3:b1","86:59:76:75:ff:cb","da:40:5d:bc:6a:ef","8a:94:01:13:ba:d2","7e:b9:97:fb:27:3b","0a:78:5c:49:34:9a","c6:52:4a:f9:86:2c","16:da:17:5e:3a:b7","fe:4a:3d:a8:9e:bb","0e:52:34:4e:eb:8d","26:8d:9d:fe:51:cd","1a:04:02:61:0b:ae","fa:ad:aa:16:32:75","6e:2d:27:b3:27:df","e2:6f:dd:81:12:ab","26:d3:42:c1:d6:3a","6e:dd:14:de:e1:e8","4e:e3:7c:b6:ae:30","b6:39:54:b7:9b:1e","e6:55:90:80:54:b9","52:2e:1c:1b:1f:41"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"1a018e03a49f4bfc904c69b0d6c08959"}}}
2020-08-13T09:15:11.427Z INFO [beat] instance/beat.go:1028 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-08-13T09:15:09.230Z"}}}
2020-08-13T09:15:11.428Z INFO instance/beat.go:310 Setup Beat: filebeat; Version: 7.8.1
2020-08-13T09:15:11.428Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.8.1' as ILM is enabled.
2020-08-13T09:15:11.428Z INFO eslegclient/connection.go:99 elasticsearch url: https://elasticsearch-es-http.elastic-system.svc:9200
2020-08-13T09:15:11.428Z INFO [publisher] pipeline/module.go:113 Beat name: ip-172-26-25-220.clickatell.com
2020-08-13T09:15:11.429Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-08-13T09:15:11.429Z INFO kibana/client.go:118 Kibana url: https://kibana-kb-http.elastic-system.svc:5601
2020-08-13T09:15:16.429Z INFO kibana/client.go:118 Kibana url: https://kibana-kb-http.elastic-system.svc:5601
2020-08-13T09:15:41.430Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":73}},"total":{"ticks":710,"time":{"ms":713},"value":710},"user":{"ticks":640,"time":{"ms":640}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"20e72410-bf57-4a12-8b7c-f4194fb73666","uptime":{"ms":30897}},"memstats":{"gc_next":16818544,"memory_alloc":13059288,"memory_total":117473240,"rss":76005376},"runtime":{"goroutines":16}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":16},"load":{"1":0.73,"15":1.07,"5":1.03,"norm":{"1":0.0456,"15":0.0669,"5":0.0644}}}}}}
2020-08-13T09:16:11.430Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":13}},"total":{"ticks":790,"time":{"ms":92},"value":790},"user":{"ticks":710,"time":{"ms":79}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"20e72410-bf57-4a12-8b7c-f4194fb73666","uptime":{"ms":60898}},"memstats":{"gc_next":15043840,"memory_alloc":8781912,"memory_total":127040312},"runtime":
2020-08-13T09:16:23.112Z INFO instance/beat.go:817 Kibana dashboards successfully loaded.
2020-08-13T09:16:23.112Z INFO instance/beat.go:463 filebeat start running.
2020-08-13T09:16:23.112Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-08-13T09:16:23.112Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-08-13T09:16:23.112Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2020-08-13T09:16:23.112Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2020-08-13T09:16:23.114Z INFO [autodiscover.pod] kubernetes/util.go:79 kubernetes: Using node ip-172-26-25-220.clickatell.com provided in the config
2020-08-13T09:16:23.114Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager