# Filebeat Pod not working properly
**URL:** https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641
**Category:** Beats
**Tags:** docker, filebeat, metricbeat
**Created:** [February 8, 2021, 5:52pm UTC](https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641 "2021-02-08T17:52:38Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@yimengael](https://discuss.elastic.co/u/yimengael)
#### Post date: [February 8, 2021, 5:52pm UTC](https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641/1 "2021-02-08T17:52:38Z")
Hi Filebeat experts,
I deployed Filebeat on my Kubernetes cluster using deamon set. I have many nodes in my cluster. In each node of my cluster, I have a Filebeat pods that is running and collecting all the pods logs on that node and push them to Kibana.
1. I noticed that for some node, Filebeat pod is working fine and collecting the logs properly. I can see that based on the following logs that is reading 2 logs files currently.
```auto
{"level":"info","timestamp":"2021-02-08T14:29:14.116Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":52680},"total":{"ticks":160400,"time":{"ms":10},"value":160400},"user":{"ticks":107720,"time":{"ms":10}}},"info":{"ephemeral_id":"d41b232b-867d-4d8d-b111-be8a5590d1a4","uptime":{"ms":488940103}},"memstats":{"gc_next":4194304,"memory_alloc":1852552,"memory_total":31294830272}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1,"batches":1,"total":1},"read":{"bytes":35},"write":{"bytes":974}},"pipeline":{"clients":4,"events":{"active":0,"published":1,"total":1},"queue":{"acked":1}}},"registrar":{"states":{"current":4,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
```
1. But in some other node in the cluster, Filebeat pod is not collecting logs and keep rebooting.I noticed in the logs that there is no harvester reading logs files as show below
```auto
{"level":"info","timestamp":"2021-02-05T22:11:08.499Z","logger":"monitoring","caller":"log/log.go:141","message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":2310,"time":{"ms":376}},"total":{"ticks":17790,"time":{"ms":2968},"value":17790},"user":{"ticks":15480,"time":{"ms":2592}}},"info":{"ephemeral_id":"a09ef099-1762-4ea4-b61c-efd8aff99f64","uptime":{"ms":180191}},"memstats":{"gc_next":10846512,"memory_alloc":7585384,"memory_total":2297070640,"rss":270336}},"filebeat":{"events":{"added":223,"done":223},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":1,"filtered":223,"total":223}}},"registrar":{"states":{"current":3889,"update":223},"writes":{"success":223,"total":223}},"system":{"load":{"1":0.22,"15":0.58,"5":0.47,"norm":{"1":0.055,"15":0.145,"5":0.1175}}}}}}
```
Also I noticed that after the above event in the logs is appearing `19 times`, Filebeat pod reboots automatically.
I just want to add that the configuration of the Filebeat pod is the same for all of them because it is a deamonset, Filebeat pod is created on kubernetes node with the same config.
So my question is that, what could be causing some Filebeat pods to read the log files correctly and some other Filebeat pods not reading the log files and just keep rebooting ?
I look forward to reading from you.
Thanks
Gael
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 9, 2021, 2:28pm UTC](https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641/2 "2021-02-09T14:28:17Z")
Hi!
Could you please run in debug mode and check/share logs for any errors?
Also can you make sure that `/var/log/` directory inside Filebeat containers contains log files?
---
### Author:  [@yimengael](https://discuss.elastic.co/u/yimengael)
#### Post date: [February 9, 2021, 5:02pm UTC](https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641/3 "2021-02-09T17:02:44Z")
@ChrsMark Here are the results for the content of folders
in the folder called `/var/log/pods`, below you have some logs there
```auto
bash-4.4# pwd/var/log/pods/default_ingress-event-responder-fabian-us-deployment-6448495b57-xpt2v_95289347-a5f6-4f23-b0d5-94bb0a4bc587/ingress-event-responder-fabian-us
bash-4.4# ls -l
total 0
lrwxrwxrwx 1 root root 165 Jan 14 21:11 0.log -> /var/lib/docker/containers/9959de3182f2e5a69eb31abe30c72775356487bc37adca5e168ae2bff465288f/9959de3182f2e5a69eb31abe30c72775356487bc37adca5e168ae2bff465288f-json.log
bash-4.4#
```
Below os the content of the `var/log/pods` folder :
```auto
bash-4.4# pwd/var/log/pods
bash-4.4# ls -l
total 0
drwxr-xr-x 3 root root 47 Jan 14 21:11 default_ingress-event-responder-fabian-us-deployment-6448495b57-xpt2v_95289347-a5f6-4f23-b0d5-94bb0a4bc587
drwxr-xr-x 3 root root 36 Feb 5 21:59 default_kube-logging-fabian-us-2st6d_2706e758-e8b1-47e2-b21f-1fbc1de3052a
drwxr-xr-x 3 root root 22 Jul 15 2020 kube-system_aws-node-9mlrn_5ec97dc2-dc13-4a54-b674-d7d6174fdf0d
drwxr-xr-x 3 root root 21 Jul 15 2020 kube-system_coredns-86d5cbb4bd-2vt4c_e5a0d207-3588-4df3-89fd-a3df3dd25a67
drwxr-xr-x 3 root root 24 Jul 15 2020 kube-system_kube-proxy-v5692_32fd3509-dbe6-4f7b-b6b8-9b26445fc132bash-4.4#
```
Below here is the config we have currently:
```auto
config
inputs:
enabled: true
path: /filebeat/inputs.d
items:
containers:
type: docker
close_inactive: 8640h
harvester_buffer_size: 20480
combine_partial: true
ignore_parsing_error: true
tail_files: false
symlinks: true
backoff: 3s
containers:
path: /var/log/containers
ids:
- "*"
processors:
- extract_kubernetes_metadata:
kubernetes:
type: log
close_inactive: 8640h
backoff: 3s
paths:
- /var/log/kubernetes/*
symlinks: true
tail_files: false
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
livenessProbe:
exec:
command:
- /filebeat/healthz.sh
initialDelaySeconds: 90
periodSeconds: 300
failureThreshold: 2
volumes:
- name: docker-containers
hostPath: /var/lib/docker/containers
mountPath: /var/lib/docker/containers
readOnly: true
- name: containers
hostPath: /var/log/containers
mountPath: /var/log/containers/pods
readOnly: true
- name: pods
hostPath: /var/log/pods
mountPath: /var/log/pods
readOnly: true
- name: kubernetes
hostPath: /var/log/kubernetes
mountPath: /var/log/kubernetes
readOnly: true
- name: filebeat-kubernetes
hostPath: /var/lib/filebeat-kubernetes
mountPath: /var/filebeat
readOnly: false
```
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 10, 2021, 11:28am UTC](https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641/4 "2021-02-10T11:28:33Z")
Hi!
What version of filebeat do you use? Your configuration looks really strange to me I have to admit. Specially this `extract_kubernetes_metadata`, it's first time I see this processor, I only know of `add_kubernetes_metadata`.
Here is a complete example of Filebeat on k8s to collect containers' logs:
>
Do you think the above could cover your case?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 10, 2021, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-pod-not-working-properly/263641/5 "2021-03-10T13:29:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.