How does the setup of filebeat on kubernetes work?

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I'm using the elastic operator "docker.elastic.co/eck/eck-operator:1.5.0" and trying to get a working filebeat running.

Somehow I don't get index templates properly set up and also ilm.

My deployment looks like this:

    apiVersion: beat.k8s.elastic.co/v1beta1
    kind: Beat
    metadata:
      name: filebeat
    spec:
      type: filebeat
      version: 7.12.1
      elasticsearchRef:
        name: quickstart
      kibanaRef:
        name: quickstart
      config:
        setup.template.enabled: true
        setup.template.overwrite: true
        filebeat:
          autodiscover:
            providers:
            - type: kubernetes
              node: ${NODE_NAME}
              hints:
                enabled: true
                default_config:
                  type: container
                  paths:
                  - /var/log/containers/*.log
        processors:
        - add_cloud_metadata: {}
        - add_host_metadata: {}
      daemonSet:
        podTemplate:
          spec:
            serviceAccountName: filebeat
            automountServiceAccountToken: true
            terminationGracePeriodSeconds: 30
            dnsPolicy: ClusterFirstWithHostNet
            hostNetwork: true # Allows to provide richer host metadata
            containers:
            - name: filebeat
              securityContext:
                runAsUser: 0
                # If using Red Hat OpenShift uncomment this:
                #privileged: true
              volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
              env:
                - name: NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
            volumes:
            - name: varlogcontainers
              hostPath:
                path: /var/log/containers
            - name: varlogpods
              hostPath:
                path: /var/log/pods
            - name: varlibdockercontainers
              hostPath:
                path: /var/lib/docker/containers
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: filebeat
    rules:
    - apiGroups: [""] # "" indicates the core API group
      resources:
      - namespaces
      - pods
      - nodes
      verbs:
      - get
      - watch
      - list
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: filebeat
      namespace: elastic-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: filebeat
    subjects:
    - kind: ServiceAccount
      name: filebeat
      namespace: elastic-system
    roleRef:
      kind: ClusterRole
      name: filebeat
      apiGroup: rbac.authorization.k8s.io

In the logs I cannot see anything suspicious. After a while it it starts running without errors. When I look into kibana, I don't see no index, no index template and also no ILM Policy.

With metricbeat everything worked just fine...

2

Can you try using the following config instead:

        default_config:
          type: container
          paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

Following Autodiscover | Filebeat Reference [7.12] | Elastic we have to be careful with autodiscover to match the path to the scope of the autodiscover condition.

3

I deleted my filebeat deployment and applied it with your suggested changes.

Sadly the result is the same.

Here is how the logs look like:

$ kubectl logs filebeat-beat-filebeat-lv72x
2021-05-19T13:42:06.970Z        INFO    instance/beat.go:660    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-05-19T13:42:06.970Z        INFO    instance/beat.go:668    Beat ID: 24aa1688-2dd2-446b-8205-96873d161166
2021-05-19T13:42:06.972Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2021-05-19T13:42:06.972Z        INFO    [beat]  instance/beat.go:996    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "24aa1688-2dd2-446b-8205-96873d161166"}}}
2021-05-19T13:42:06.972Z        INFO    [beat]  instance/beat.go:1005   Build info      {"system_info": {"build": {"commit": "651a2ad1225f3d4420a22eba847de385b71f711d", "libbeat": "7.12.1", "time": "2021-04-20T20:58:32.000Z", "version": "7.12.1"}}}
2021-05-19T13:42:06.972Z        INFO    [beat]  instance/beat.go:1008   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.15.9"}}}
2021-05-19T13:42:06.973Z        INFO    [beat]  instance/beat.go:1012   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-05-03T13:04:46Z","containerized":true,"name":"b0b52623-2a74-4674-808a-7c757651f054","ip":CENSORED,"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"5ea6a666bf6bfd4b2167796371a02dc5"}}}
2021-05-19T13:42:06.973Z        INFO    [beat]  instance/beat.go:1041   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 6, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-05-19T13:42:05.300Z"}}}
2021-05-19T13:42:06.973Z        INFO    instance/beat.go:304    Setup Beat: filebeat; Version: 7.12.1
2021-05-19T13:42:06.973Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.12.1' as ILM is enabled.
2021-05-19T13:42:07.067Z        WARN    [cfgwarn]       tlscommon/config.go:101 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are 
present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-05-19T13:42:07.067Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://quickstart-es-http.elastic-system.svc:9200
2021-05-19T13:42:07.068Z        INFO    [publisher]     pipeline/module.go:113  Beat name: b0b52623-2a74-4674-808a-7c757651f054
2021-05-19T13:42:07.069Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-05-19T13:42:07.069Z        WARN    [cfgwarn]       tlscommon/config.go:101 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are 
present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-05-19T13:42:07.069Z        INFO    kibana/client.go:119    Kibana url: https://quickstart-kb-http.elastic-system.svc:5601
2021-05-19T13:42:09.971Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2021-05-19T13:42:26.072Z        WARN    [cfgwarn]       tlscommon/config.go:101 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are 
present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-05-19T13:42:26.267Z        INFO    kibana/client.go:119    Kibana url: https://quickstart-kb-http.elastic-system.svc:5601
2021-05-19T13:42:37.071Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":10000}},"id":"4a4c0667aeea6d8c23f32269152c505d556bc02826bc198a2f9f756aa4370a1a","stats":{"periods":246,"throttled":{"ns":36634310287,"periods":198}}},"cpuacct":{"id":"4a4c0667aeea6d8c23f32269152c505d556bc02826bc198a2f9f756aa4370a1a","total":{"ns":2257821083}},"memory":{"id":"4a4c0667aeea6d8c23f32269152c505d556bc02826bc198a2f9f756aa4370a1a","mem":{"limit":{"bytes":209715200},"usage":{"bytes":50806784}}}},"cpu":{"system":{"ticks":260,"time":{"ms":262}},"total":{"ticks":2240,"time":{"ms":2248},"value":2240},"user":{"ticks":1980,"time":{"ms":1986}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"7282807e-aff7-453b-b936-7f8ebe3471ea","uptime":{"ms":30802}},"memstats":{"gc_next":31585616,"memory_alloc":26562608,"memory_sys":148426760,"memory_total":425039552,"rss":97169408},"runtime":{"goroutines":15}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.27,"15":0.43,"5":0.39,"norm":{"1":0.0338,"15":0.0538,"5":0.0488}}}}}}
2021-05-19T13:43:07.071Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"stats":{"periods":56}},"cpuacct":{"total":{"ns":100036279}},"memory":{"mem":{"usage":{"bytes":-8753152}}}},"cpu":{"system":{"ticks":310,"time":{"ms":56}},"total":{"ticks":2330,"time":{"ms":97},"value":2330},"user":{"ticks":2020,"time":{"ms":41}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"7282807e-aff7-453b-b936-7f8ebe3471ea","uptime":{"ms":60800}},"memstats":{"gc_next":24073296,"memory_alloc":16617064,"memory_total":433020920,"rss":87932928},"runtime":{"goroutines":15}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.16,"15":0.42,"5":0.36,"norm":{"1":0.02,"15":0.0525,"5":0.045}}}}}}
4

I tried another deployment. This time without autodiiscover. Still no luck... I start to think that ilm setup is broken in filebeat 7.12.1

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.