Filebeat Pod not working properly

yimengael · February 8, 2021, 5:52pm

Hi Filebeat experts,

I deployed Filebeat on my Kubernetes cluster using deamon set. I have many nodes in my cluster. In each node of my cluster, I have a Filebeat pods that is running and collecting all the pods logs on that node and push them to Kibana.

I noticed that for some node, Filebeat pod is working fine and collecting the logs properly. I can see that based on the following logs that is reading 2 logs files currently.

{"level":"info","timestamp":"2021-02-08T14:29:14.116Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":52680},"total":{"ticks":160400,"time":{"ms":10},"value":160400},"user":{"ticks":107720,"time":{"ms":10}}},"info":{"ephemeral_id":"d41b232b-867d-4d8d-b111-be8a5590d1a4","uptime":{"ms":488940103}},"memstats":{"gc_next":4194304,"memory_alloc":1852552,"memory_total":31294830272}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1,"batches":1,"total":1},"read":{"bytes":35},"write":{"bytes":974}},"pipeline":{"clients":4,"events":{"active":0,"published":1,"total":1},"queue":{"acked":1}}},"registrar":{"states":{"current":4,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}

But in some other node in the cluster, Filebeat pod is not collecting logs and keep rebooting.I noticed in the logs that there is no harvester reading logs files as show below

{"level":"info","timestamp":"2021-02-05T22:11:08.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":2310,"time":{"ms":376}},"total":{"ticks":17790,"time":{"ms":2968},"value":17790},"user":{"ticks":15480,"time":{"ms":2592}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":180191}},"memstats":{"gc_next":10846512,"memory_alloc":7585384,"memory_total":2297070640,"rss":270336}},"filebeat":{"events":{"added":223,"done":223},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":223,"total":223}}},"registrar":{"states":{"current":3889,"update":223},"writes":{"success":223,"total":223}},"system":{"load":{"1":0.22,"15":0.58,"5":0.47,"norm":{"1":0.055,"15":0.145,"5":0.1175}}}}}}

Also I noticed that after the above event in the logs is appearing 19 times, Filebeat pod reboots automatically.
I just want to add that the configuration of the Filebeat pod is the same for all of them because it is a deamonset, Filebeat pod is created on kubernetes node with the same config.

So my question is that, what could be causing some Filebeat pods to read the log files correctly and some other Filebeat pods not reading the log files and just keep rebooting ?

I look forward to reading from you.
Thanks
Gael

ChrsMark · February 9, 2021, 2:28pm

Hi!

Could you please run in debug mode and check/share logs for any errors?
Also can you make sure that /var/log/ directory inside Filebeat containers contains log files?

yimengael · February 9, 2021, 5:02pm

@ChrsMark Here are the results for the content of folders
in the folder called /var/log/pods, below you have some logs there

bash-4.4# pwd/var/log/pods/default_ingress-event-responder-fabian-us-deployment-6448495b57-xpt2v_95289347-a5f6-4f23-b0d5-94bb0a4bc587/ingress-event-responder-fabian-us
bash-4.4# ls -l
total 0
lrwxrwxrwx    1 root     root           165 Jan 14 21:11 0.log -> /var/lib/docker/containers/9959de3182f2e5a69eb31abe30c72775356487bc37adca5e168ae2bff465288f/9959de3182f2e5a69eb31abe30c72775356487bc37adca5e168ae2bff465288f-json.log
bash-4.4#

Below os the content of the var/log/pods folder :

bash-4.4# pwd/var/log/pods
bash-4.4# ls -l
total 0
drwxr-xr-x    3 root     root            47 Jan 14 21:11 default_ingress-event-responder-fabian-us-deployment-6448495b57-xpt2v_95289347-a5f6-4f23-b0d5-94bb0a4bc587
drwxr-xr-x    3 root     root            36 Feb  5 21:59 default_kube-logging-fabian-us-2st6d_2706e758-e8b1-47e2-b21f-1fbc1de3052a
drwxr-xr-x    3 root     root            22 Jul 15  2020 kube-system_aws-node-9mlrn_5ec97dc2-dc13-4a54-b674-d7d6174fdf0d
drwxr-xr-x    3 root     root            21 Jul 15  2020 kube-system_coredns-86d5cbb4bd-2vt4c_e5a0d207-3588-4df3-89fd-a3df3dd25a67
drwxr-xr-x    3 root     root            24 Jul 15  2020 kube-system_kube-proxy-v5692_32fd3509-dbe6-4f7b-b6b8-9b26445fc132bash-4.4#

Below here is the config we have currently:

config
  inputs:
    enabled: true
    path: /filebeat/inputs.d
    items:
      containers:
        type: docker
        close_inactive: 8640h
        harvester_buffer_size: 20480
        combine_partial: true
        ignore_parsing_error: true
        tail_files: false
        symlinks: true
        backoff: 3s
        containers:
          path: /var/log/containers
          ids:
            - "*"
        processors:
          - extract_kubernetes_metadata:
      kubernetes:
        type: log
        close_inactive: 8640h
        backoff: 3s
        paths:
          - /var/log/kubernetes/*
        symlinks: true
        tail_files: false

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 25%

livenessProbe:
  exec:
    command:
    - /filebeat/healthz.sh
  initialDelaySeconds: 90
  periodSeconds: 300
  failureThreshold: 2

volumes:
  - name: docker-containers
    hostPath: /var/lib/docker/containers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: containers
    hostPath: /var/log/containers
    mountPath: /var/log/containers/pods
    readOnly: true
  - name: pods
    hostPath: /var/log/pods
    mountPath: /var/log/pods
    readOnly: true
  - name: kubernetes
    hostPath: /var/log/kubernetes
    mountPath: /var/log/kubernetes
    readOnly: true
  - name: filebeat-kubernetes
    hostPath: /var/lib/filebeat-kubernetes
    mountPath: /var/filebeat
    readOnly: false

ChrsMark · February 10, 2021, 11:28am

Hi!

What version of filebeat do you use? Your configuration looks really strange to me I have to admit. Specially this extract_kubernetes_metadata, it's first time I see this processor, I only know of add_kubernetes_metadata.

Here is a complete example of Filebeat on k8s to collect containers' logs:

github.com

elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

This file has been truncated. show original

Do you think the above could cover your case?

system · March 10, 2021, 1:29pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat K8S pod keeps restarting and Logs stop getting to Kibana Beats docker	4	785	March 9, 2021
Filebeat does not harvest logs Beats filebeat	5	1543	February 23, 2019
Filebeat on kubernetes not pulling logs from all pods Beats filebeat	1	631	July 29, 2020
Filebeat adding incorrect kubernetes metadata to events going to logstash Beats filebeat	5	743	April 2, 2024
Filebeat is partially collecting logs Beats filebeat	3	810	May 9, 2019

Filebeat Pod not working properly

Related Topics