I have deploy filebeat in kubernetes as daemon set (the node is ubuntu 18.04), but some of the filebeat pod do not send any log, it did not harvest any log. What might cause this?
This is my filebeat configuration:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.labels.project: myproject
config:
- type: docker
containers.ids:
- "${data.kubernetes.container.id}"
multiline.pattern: '^[0-9]'
multiline.negate: true
multiline.match: after
- condition:
equals:
kubernetes.namespace: kube-system
config:
- type: docker
containers.ids:
- "${data.kubernetes.container.id}"
processors:
- add_cloud_metadata:
output.file:
path: "/var/log/kubernetes"
filename: kubernetes.log
rotate_every_kb: 500000
in filebeat that ships log I got this message from its log:
2019-01-24T02:01:48.747Z INFO log/harvester.go:254 **Harvester started for file: /var/lib/docker/containers/bc4c0cd1dff53c1eec4767be7c35cbb4fdf2a9fa41524d2201977df33a8ade35**/bc4c0cd1dff53c1eec4767be7c35cbb4fdf2a9fa41524d2201977df33a8ade35-json.log
2019-01-24T02:02:07.463Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":24270,"time":{"ms":12}},"total":{"ticks":98840,"time":{"ms":20},"value":98840},"user":{"ticks":74570,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"a0258a82-20f9-48d9-bc69-ebf81d41dd91","uptime":{"ms":56130025}},"memstats":{"gc_next":11523040,"memory_alloc":8927368,"memory_total":14386395872,"rss":-311296}},"filebeat":{"events":{"added":37,"done":37},"harvester":{"open_files":4,"running":4,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":36,"batches":9,"total":36},"write":{"bytes":40906}},"pipeline":{"clients":121,"events":{"active":0,"filtered":1,"published":36,"total":37},"queue":{"acked":36}}},"registrar":{"states":{"current":26,"update":37},"writes":{"success":10,"total":10}},"system":{"load":{"1":0.08,"15":0.1,"5":0.12,"norm":{"1":0.04,"15":0.05,"5":0.06}}}}}}
In filebeat that doesn't i only see
2019-01-24T02:28:56.359Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1100},"total":{"ticks":4170,"time":{"ms":4},"value":4170},"user":{"ticks":3070,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"91e54eb1-21f2-4f67-9734-96749f3a05d4","uptime":{"ms":57780020}},"memstats":{"gc_next":4194304,"memory_alloc":1881584,"memory_total":486681984}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.06,"5":0.12,"norm":{"1":0.05,"15":0.03,"5":0.06}}}}}}
).