Filebeat K8S pod keeps restarting and Logs stop getting to Kibana

Elastic Stack Beats
1

Hello,
I have configured Filebeat on my Kubernetes cluster and it is running in a kubernetes pod. Sometimes I noticed that Filebeat stopped collecting logs and the pods where filebeat is installed keeps restarting.
I investigated and I noticed that there is no error logs on my filebeat pod and also there is no error logs on the previous pod logs.
Could you please help investigate what is the issue ? why Filebeat stopped collecting logs ? Is there a config issue ?

Below is my Filebeat pod logs:

    {"level":"info","timestamp":"2021-02-05T22:08:08.400Z","caller":"instance/beat.go:544","message":"Home path: [/filebeat] Config path: [/filebeat] Data path: [/var/filebeat/data] Logs path: [/var/filebeat/logs]"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.400Z","caller":"instance/beat.go:551","message":"Beat UUID: ed79b49a-9697-4651-be96-ffac9b2f2441"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.400Z","logger":"seccomp","caller":"seccomp/seccomp.go:116","message":"Syscall filter successfully installed"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.400Z","logger":"beat","caller":"instance/beat.go:768","message":"Beat info","system_info":{"beat":{"path":{"config":"/filebeat","data":"/var/filebeat/data","home"
    :"/filebeat","logs":"/var/filebeat/logs"},"type":"filebeat","uuid":"ed79b49a-9697-4651-be96-ffac9b2f2441"}}}
    {"level":"info","timestamp":"2021-02-05T22:08:08.400Z","logger":"beat","caller":"instance/beat.go:777","message":"Build info","system_info":{"build":{"commit":"unknown","libbeat":"6.4.2","time":"1754-08-30T22:43
    :41.128Z","version":"6.4.2"}}}
    {"level":"info","timestamp":"2021-02-05T22:08:08.400Z","logger":"beat","caller":"instance/beat.go:780","message":"Go runtime info","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.11
    "}}}
    {"level":"info","timestamp":"2021-02-05T22:08:08.401Z","logger":"beat","caller":"instance/beat.go:784","message":"Host info","system_info":{"host":{"architecture":"x86_64","boot_time":"2020-08-12T23:44:48Z","con
    tainerized":true,"hostname":"kube-logging-fabian-us-tb96x","ips":["127.0.0.1/8","10.70.15.15/32"],"kernel_version":"4.14.181-142.260.amzn2.x86_64","mac_addresses":["26:b0:e2:84:0d:5b"],"os":{"family":"","platfor
    m":"alpine","name":"Alpine Linux","version":"","major":0,"minor":0,"patch":0},"timezone":"UTC","timezone_offset_sec":0}}}
    {"level":"info","timestamp":"2021-02-05T22:08:08.402Z","logger":"beat","caller":"instance/beat.go:813","message":"Process info","system_info":{"process":{"capabilities":{"inheritable":["chown","dac_override","fo
    wner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpca
    p","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","m
    knod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"
    cwd":"/var/filebeat","exe":"/filebeat/filebeat","name":"filebeat","pid":1,"ppid":0,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2021-02-05T22:08:07.240Z"}}}
    {"level":"info","timestamp":"2021-02-05T22:08:08.402Z","caller":"instance/beat.go:273","message":"Setup Beat: filebeat; Version: 6.4.2"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.402Z","caller":"pipeline/module.go:98","message":"Beat name: kube-logging-fabian-us-tb96x"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.497Z","logger":"monitoring","caller":"log/log.go:114","message":"Starting metrics logging every 30s"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.497Z","caller":"instance/beat.go:367","message":"filebeat start running."}
    {"level":"warn","timestamp":"2021-02-05T22:08:08.498Z","logger":"cfgwarn","caller":"api/server.go:34","message":"EXPERIMENTAL: Metrics endpoint is enabled."}
    {"level":"info","timestamp":"2021-02-05T22:08:08.498Z","caller":"api/server.go:38","message":"Starting stats endpoint"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.498Z","caller":"registrar/registrar.go:134","message":"Loading registrar data from /var/filebeat/data/registry"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.498Z","caller":"api/server.go:49","message":"Metrics endpoint listening on: localhost:5066"}
    {"level":"info","timestamp":"2021-02-05T22:08:08.999Z","caller":"registrar/registrar.go:141","message":"States Loaded from registrar: 3889"}
    {"level":"warn","timestamp":"2021-02-05T22:08:08.999Z","caller":"beater/filebeat.go:371","message":"Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch outpu
    t is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning."}
    {"level":"info","timestamp":"2021-02-05T22:08:09.000Z","caller":"crawler/crawler.go:72","message":"Loading Inputs: 0"}
    {"level":"warn","timestamp":"2021-02-05T22:08:09.000Z","logger":"cfgwarn","caller":"docker/input.go:46","message":"EXPERIMENTAL: Docker input is enabled."}
    {"level":"info","timestamp":"2021-02-05T22:08:38.500Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":410,"
    time":{"ms":419}},"total":{"ticks":3000,"time":{"ms":3010},"value":3000},"user":{"ticks":2590,"time":{"ms":2591}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":30191}},"memstats":
    {"gc_next":14029664,"memory_alloc":12822024,"memory_total":355838112,"rss":36261888}},"filebeat":{"events":{"active":5,"added":233,"done":228},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"modu
    le":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":1,"filtered":232,"total":233}}},"registrar":{"states":{"current":3889,"update":229},"writes":{"success":228,"total":229}
    },"system":{"cpu":{"cores":4},"load":{"1":0.84,"15":0.64,"5":0.62,"norm":{"1":0.21,"15":0.16,"5":0.155}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:09:08.498Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":820,"
    time":{"ms":401}},"total":{"ticks":5940,"time":{"ms":2940},"value":5940},"user":{"ticks":5120,"time":{"ms":2539}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":60192}},"memstats":
    {"gc_next":13935552,"memory_alloc":11284232,"memory_total":697433808}},"filebeat":{"events":{"added":232,"done":232},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeli
    ne":{"clients":1,"events":{"active":1,"filtered":232,"total":232}}},"registrar":{"states":{"current":3889,"update":232},"writes":{"success":232,"total":232}},"system":{"load":{"1":0.56,"15":0.62,"5":0.58,"norm":
    {"1":0.14,"15":0.155,"5":0.145}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:09:38.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1240,
    "time":{"ms":422}},"total":{"ticks":8900,"time":{"ms":2961},"value":8900},"user":{"ticks":7660,"time":{"ms":2539}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":90191}},"memstats"
    :{"gc_next":14391584,"memory_alloc":8198856,"memory_total":1054639360,"rss":536576}},"filebeat":{"events":{"added":233,"done":233},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"runnin
    g":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":233,"total":233}}},"registrar":{"states":{"current":3889,"update":233},"writes":{"success":233,"total":233}},"system":{"load":{"1":0.4,"15":0.61,"5"
    :0.54,"norm":{"1":0.1,"15":0.1525,"5":0.135}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:10:08.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1580,
    "time":{"ms":338}},"total":{"ticks":11870,"time":{"ms":2968},"value":11870},"user":{"ticks":10290,"time":{"ms":2630}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":120191}},"memst
    ats":{"gc_next":16688896,"memory_alloc":8351376,"memory_total":1452268944}},"filebeat":{"events":{"added":230,"done":230},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"p
    ipeline":{"clients":1,"events":{"active":1,"filtered":230,"total":230}}},"registrar":{"states":{"current":3889,"update":230},"writes":{"success":230,"total":230}},"system":{"load":{"1":0.6,"15":0.62,"5":0.58,"no
    rm":{"1":0.15,"15":0.155,"5":0.145}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:10:38.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1930,
    "time":{"ms":359}},"total":{"ticks":14820,"time":{"ms":2955},"value":14820},"user":{"ticks":12890,"time":{"ms":2596}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":150192}},"memst
    ats":{"gc_next":16690208,"memory_alloc":14666504,"memory_total":1885048520,"rss":270336}},"filebeat":{"events":{"added":227,"done":227},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"r
    unning":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":227,"total":227}}},"registrar":{"states":{"current":3889,"update":227},"writes":{"success":227,"total":227}},"system":{"load":{"1":0.36,"15":0.6,"5":0.52,"norm":{"1":0.09,"15":0.15,"5":0.13}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:11:08.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":2310,"time":{"ms":376}},"total":{"ticks":17790,"time":{"ms":2968},"value":17790},"user":{"ticks":15480,"time":{"ms":2592}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":180191}},"memstats":{"gc_next":10846512,"memory_alloc":7585384,"memory_total":2297070640,"rss":270336}},"filebeat":{"events":{"added":223,"done":223},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":223,"total":223}}},"registrar":{"states":{"current":3889,"update":223},"writes":{"success":223,"total":223}},"system":{"load":{"1":0.22,"15":0.58,"5":0.47,"norm":{"1":0.055,"15":0.145,"5":0.1175}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:11:38.498Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":2750,"time":{"ms":438}},"total":{"ticks":20750,"time":{"ms":2955},"value":20750},"user":{"ticks":18000,"time":{"ms":2517}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":210191}},"memstats":{"gc_next":16899744,"memory_alloc":15020328,"memory_total":2660059120}},"filebeat":{"events":{"added":227,"done":227},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":227,"total":227}}},"registrar":{"states":{"current":3889,"update":227},"writes":{"success":227,"total":227}},"system":{"load":{"1":0.21,"15":0.57,"5":0.44,"norm":{"1":0.0525,"15":0.1425,"5":0.11}}}}}}
    {"level":"info","timestamp":"2021-02-05T22:12:08.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":3100,"time":{"ms":352}},"total":{"ticks":23700,"time":{"ms":2955},"value":23700},"user":{"ticks":20600,"time":{"ms":2603}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":240191}},"memstats":{"gc_next":14793664,"memory_alloc":9621960,"memory_total":3113947416,"rss":1617920}},"filebeat":{"events":{"added":215,"done":215},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":215,"total":215}}},"registrar":{"states":{"current":3889,"update":215},"writes":{"success":215,"total":215}},"system":{"load":{"1":0.27,"15":0.56,"5":0.43,"norm":{"1":0.0675,"15":0.14,"5":0.1075}}}}}}

Please i looked forward to hearing from you.
Thanks
Gael

2

I wanted to add more details. Here is my configuration of the input:

config:
    inputs:
        enabled: true
        path: /filebeat/inputs.d
        items:
          containers:
            type: docker
            close_inactive: 8640h
            harvester_buffer_size: 20480
            combine_partial: true
            ignore_parsing_error: true
            tail_files: false
            symlinks: true
            backoff: 3s
            containers:
              path: /var/log/containers
              ids:
                - "*"
            processors:
              - extract_kubernetes_metadata:
          kubernetes:
            type: log
            close_inactive: 8640h
            backoff: 3s
            paths:
              - /var/log/kubernetes/*
            symlinks: true
            tail_files: false
    updateStrategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 25%

    livenessProbe:
      exec:
        command:
        - /filebeat/healthz.sh
      initialDelaySeconds: 90
      periodSeconds: 300
      failureThreshold: 2

    volumes:
      - name: docker-containers
        hostPath: /var/lib/docker/containers
        mountPath: /var/lib/docker/containers
        readOnly: true
      - name: containers
        hostPath: /var/log/containers
        mountPath: /var/log/containers/pods
        readOnly: true
      - name: pods
        hostPath: /var/log/pods
        mountPath: /var/log/pods
        readOnly: true
      - name: kubernetes
        hostPath: /var/log/kubernetes
        mountPath: /var/log/kubernetes
        readOnly: true
      - name: filebeat-kubernetes
        hostPath: /var/lib/filebeat-kubernetes
        mountPath: /var/filebeat
        readOnly: false

Something that I also noticed that can help investigate is that after 19 times the following events happen, the Filebeat pod just reboots.

{"level":"info","timestamp":"2021-02-05T22:08:06.498Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":6680,"time":{"ms":346}},"total":{"ticks":56150,"time":{"ms":2971},"value":56150},"user":{"ticks":49470,"time":{"ms":2625}}},"info":{"ephemeral_id":"0305c043-2c56-48a9-ad91-262b2301441f","uptime":{"ms":570199}},"memstats":{"gc_next":10209808,"memory_alloc":7129112,"memory_total":7881510112}},"filebeat":{"events":{"added":229,"done":229},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":1,"filtered":229,"total":229}}},"registrar":{"states":{"current":3889,"update":229},"writes":{"success":229,"total":229}},"system":{"load":{"1":1.12,"15":0.65,"5":0.65,"norm":{"1":0.28,"15":0.1625,"5":0.1625}}}}}}
4

Hello, this is a question for the Beats team, not the Kibana team.

5

@Marius_Dragomir, how can I reference the Beat team to help me on this ? Thanks.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.