Filebeat harvester not starting

Hi,

I'm running Filebeat 7.8.1 on ECK, however, it seems that the harvester is not starting. I'm not getting any logs in Elasticsearch.

After some investigation, I found that the /usr/share/filebeat/data/registry/filebeat/data.json file is empty, and I suspect this is the reason it is not working. I'm not seeing any error logs on the pod.

Anyone with a similar issue or any advice?

Filebeat config:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: elastic-system
spec:
  type: filebeat
  version: 7.8.1
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${HOSTNAME}
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
    logging.level: debug
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        tolerations:
        - key: "nodetype"
          operator: "Equal"
          value: "elk"
          effect: "NoSchedule"
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - events
  - nodes
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

Pod Logs:

2020-08-13T09:15:10.928Z        INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-08-13T09:15:10.929Z        INFO    instance/beat.go:655    Beat ID: 155adddd-2cba-4a5f-bff5-ff7e22496e0d
2020-08-13T09:15:10.930Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:93     add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"xxxxxxxxxxx"},"availability_zone":"eu-west-1a","image":{"id":"ami-0de1b659c5a3fd30a"},"instance":{"id":"i-xxxxxxxxxxxxx"},"machine":{"type":"m5.4xlarge"},"provider":"aws","region":"eu-west-1"}
2020-08-13T09:15:11.227Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-08-13T09:15:11.227Z        INFO    [beat]  instance/beat.go:983    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "155adddd-2cba-4a5f-bff5-ff7e22496e0d"}}}
2020-08-13T09:15:11.227Z        INFO    [beat]  instance/beat.go:992    Build info      {"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T15:12:45.000Z", "version": "7.8.1"}}}
2020-08-13T09:15:11.227Z        INFO    [beat]  instance/beat.go:995    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.13.10"}}}
2020-08-13T09:15:11.329Z        INFO    [beat]  instance/beat.go:999    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-06-25T11:16:28Z","containerized":false,"name":"ip-172-26-25-220.clickatell.com","ip":["127.0.0.1/8","::1/128","172.26.25.220/23","fe80::cb:abff:fedd:4a54/64","172.26.25.247/23","fe80::7d:58ff:fec8:5410/64","fe80::d0f3:2bff:fee9:e623/64","fe80::80e7:4aff:fe57:f7f9/64","fe80::f842:d7ff:fe14:aa7f/64","fe80::cc9f:2dff:fe82:20cd/64","fe80::e813:47ff:fe98:6415/64","fe80::dc5a:eff:feb2:2aed/64","fe80::38bc:5cff:fedc:fe64/64","fe80::d0c4:38ff:feed:f536/64","fe80::f0d2:14ff:fec8:a8cc/64","fe80::84:5bff:feed:f93c/64","fe80::102e:b5ff:fe0a:781c/64","172.26.24.220/23","fe80::60:51ff:fe71:3e40/64","fe80::10f1:4fff:fe53:60bd/64","fe80::9484:c0ff:fe67:4edf/64","fe80::2cbd:a0ff:fe6c:7d22/64","fe80::d8a2:a6ff:fe6a:ae1d/64","fe80::ece9:d1ff:fe4b:b4f1/64","fe80::50d4:c3ff:feae:ba94/64","fe80::d85d:1ff:fe7e:7798/64","fe80::38c0:b9ff:fe25:fe60/64","fe80::807b:45ff:fef5:a4ad/64","fe80::6cf2:b9ff:fef7:a61c/64","fe80::903a:b9ff:feac:7e77/64","fe80::34dc:68ff:fedd:642b/64","fe80::30f2:a4ff:feae:97e2/64","fe80::74f9:fdff:fea0:838d/64","fe80::a0c0:eaff:fee1:576/64","fe80::f058:35ff:fe4e:d5fe/64","fe80::b06d:a7ff:fee8:bd65/64","fe80::6855:89ff:fe88:70c9/64","fe80::ca7:c7ff:fef6:14e7/64","fe80::38ac:73ff:fe29:3f40/64","fe80::282d:8aff:fee1:59a1/64","fe80::8c4:9aff:fe2b:5406/64","fe80::9cea:99ff:fe7e:2c1b/64","fe80::4008:edff:fe10:fbd0/64","fe80::7c71:6eff:fe9e:4453/64","fe80::70fa:2eff:fecb:c74c/64","fe80::ccfb:caff:fe15:6f0f/64","fe80::b8f3:dbff:fe84:9fd9/64","fe80::5806:48ff:fe86:ec9e/64","fe80::2cc2:9eff:fef5:a96c/64","fe80::1c93:e2ff:fe40:aaa5/64","fe80::bcf7:adff:fe57:1d51/64","172.26.24.200/23","fe80::d1:99ff:fe90:3532/64","fe80::74af:7eff:fe44:3669/64","fe80::10cf:beff:fe3c:c39e/64","fe80::fc6d:a3ff:fe08:b833/64","fe80::c3e:dfff:fe3c:a3b1/64","fe80::8459:76ff:fe75:ffcb/64","fe80::d840:5dff:febc:6aef/64","fe80::8894:1ff:fe13:bad2/64","fe80::7cb9:97ff:fefb:273b/64","fe80::878:5cff:fe49:349a/64","fe80::c452:4aff:fef9:862c/64","fe80::14da:17ff:fe5e:3ab7/64","fe80::fc4a:3dff:fea8:9ebb/64","fe80::c52:34ff:fe4e:eb8d/64","fe80::248d:9dff:fefe:51cd/64","fe80::1804:2ff:fe61:bae/64","fe80::f8ad:aaff:fe16:3275/64","fe80::6c2d:27ff:feb3:27df/64","fe80::e06f:ddff:fe81:12ab/64","fe80::24d3:42ff:fec1:d63a/64","fe80::6cdd:14ff:fede:e1e8/64","fe80::4ce3:7cff:feb6:ae30/64","fe80::b439:54ff:feb7:9b1e/64","fe80::e455:90ff:fe80:54b9/64","fe80::502e:1cff:fe1b:1f41/64"],"kernel_version":"4.14.181-140.257.amzn2.x86_64","mac":["02:cb:ab:dd:4a:54","02:7d:58:c8:54:10","d2:f3:2b:e9:e6:23","82:e7:4a:57:f7:f9","fa:42:d7:14:aa:7f","ce:9f:2d:82:20:cd","ea:13:47:98:64:15","de:5a:0e:b2:2a:ed","3a:bc:5c:dc:fe:64","d2:c4:38:ed:f5:36","f2:d2:14:c8:a8:cc","02:84:5b:ed:f9:3c","12:2e:b5:0a:78:1c","02:60:51:71:3e:40","12:f1:4f:53:60:bd","96:84:c0:67:4e:df","2e:bd:a0:6c:7d:22","da:a2:a6:6a:ae:1d","ee:e9:d1:4b:b4:f1","52:d4:c3:ae:ba:94","da:5d:01:7e:77:98","3a:c0:b9:25:fe:60","82:7b:45:f5:a4:ad","6e:f2:b9:f7:a6:1c","92:3a:b9:ac:7e:77","36:dc:68:dd:64:2b","32:f2:a4:ae:97:e2","76:f9:fd:a0:83:8d","a2:c0:ea:e1:05:76","f2:58:35:4e:d5:fe","b2:6d:a7:e8:bd:65","6a:55:89:88:70:c9","0e:a7:c7:f6:14:e7","3a:ac:73:29:3f:40","2a:2d:8a:e1:59:a1","0a:c4:9a:2b:54:06","9e:ea:99:7e:2c:1b","42:08:ed:10:fb:d0","7e:71:6e:9e:44:53","72:fa:2e:cb:c7:4c","ce:fb:ca:15:6f:0f","ba:f3:db:84:9f:d9","5a:06:48:86:ec:9e","2e:c2:9e:f5:a9:6c","1e:93:e2:40:aa:a5","be:f7:ad:57:1d:51","02:d1:99:90:35:32","76:af:7e:44:36:69","12:cf:be:3c:c3:9e","fe:6d:a3:08:b8:33","0e:3e:df:3c:a3:b1","86:59:76:75:ff:cb","da:40:5d:bc:6a:ef","8a:94:01:13:ba:d2","7e:b9:97:fb:27:3b","0a:78:5c:49:34:9a","c6:52:4a:f9:86:2c","16:da:17:5e:3a:b7","fe:4a:3d:a8:9e:bb","0e:52:34:4e:eb:8d","26:8d:9d:fe:51:cd","1a:04:02:61:0b:ae","fa:ad:aa:16:32:75","6e:2d:27:b3:27:df","e2:6f:dd:81:12:ab","26:d3:42:c1:d6:3a","6e:dd:14:de:e1:e8","4e:e3:7c:b6:ae:30","b6:39:54:b7:9b:1e","e6:55:90:80:54:b9","52:2e:1c:1b:1f:41"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"1a018e03a49f4bfc904c69b0d6c08959"}}}
2020-08-13T09:15:11.427Z        INFO    [beat]  instance/beat.go:1028   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-08-13T09:15:09.230Z"}}}
2020-08-13T09:15:11.428Z        INFO    instance/beat.go:310    Setup Beat: filebeat; Version: 7.8.1
2020-08-13T09:15:11.428Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.8.1' as ILM is enabled.
2020-08-13T09:15:11.428Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://elasticsearch-es-http.elastic-system.svc:9200
2020-08-13T09:15:11.428Z        INFO    [publisher]     pipeline/module.go:113  Beat name: ip-172-26-25-220.clickatell.com
2020-08-13T09:15:11.429Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-08-13T09:15:11.429Z        INFO    kibana/client.go:118    Kibana url: https://kibana-kb-http.elastic-system.svc:5601
2020-08-13T09:15:16.429Z        INFO    kibana/client.go:118    Kibana url: https://kibana-kb-http.elastic-system.svc:5601
2020-08-13T09:15:41.430Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":73}},"total":{"ticks":710,"time":{"ms":713},"value":710},"user":{"ticks":640,"time":{"ms":640}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"20e72410-bf57-4a12-8b7c-f4194fb73666","uptime":{"ms":30897}},"memstats":{"gc_next":16818544,"memory_alloc":13059288,"memory_total":117473240,"rss":76005376},"runtime":{"goroutines":16}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":16},"load":{"1":0.73,"15":1.07,"5":1.03,"norm":{"1":0.0456,"15":0.0669,"5":0.0644}}}}}}
2020-08-13T09:16:11.430Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":13}},"total":{"ticks":790,"time":{"ms":92},"value":790},"user":{"ticks":710,"time":{"ms":79}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"20e72410-bf57-4a12-8b7c-f4194fb73666","uptime":{"ms":60898}},"memstats":{"gc_next":15043840,"memory_alloc":8781912,"memory_total":127040312},"runtime":
2020-08-13T09:16:23.112Z        INFO    instance/beat.go:817    Kibana dashboards successfully loaded.
2020-08-13T09:16:23.112Z        INFO    instance/beat.go:463    filebeat start running.
2020-08-13T09:16:23.112Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-08-13T09:16:23.112Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2020-08-13T09:16:23.112Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 0
2020-08-13T09:16:23.112Z        INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 0
2020-08-13T09:16:23.114Z        INFO    [autodiscover.pod]      kubernetes/util.go:79   kubernetes: Using node ip-172-26-25-220.clickatell.com provided in the config
2020-08-13T09:16:23.114Z        INFO    [autodiscover]  autodiscover/autodiscover.go:113        Starting autodiscover manager

Could you please indent the configuration as it is done in the original config file?

Hi. Apologies. Updated with indentations.

Hi,

According to your configuration we should have some logs at the debug level :

It is surprising to see only messages at the INFO level. Could you check that the manifest you provided is actually the one you are using ?

I just gave it a try on EKS (which I guess is your platform given the beat name) and it is working as expected.

Hi,

Yes, running on EKS. I had to trim them yesterday as it was to big, so it was not the full pod log output. You can see full pod logs here: https://pastebin.com/GNWLNczW

I'm not able to reproduce your issue on EKS (v1.16.13 / CentOS Linux 4.14.186-146.268.amzn2.x86_64).

Do you have any security policy enforcement in place ? (PSP, SELinux ....)

Hi. No, did not have any security pollcy in place.

I ended up deleting everything (including the ECK operator) and redeployed. Thereafter it started to work.