Hi guys. Need help in that case. Have some trouble with deploying daemonset of filebeat's (releases 7.12 or 7.11.2) in openshift cluster by runtime cri. After starting container, filebeat open harvesters for finded files, read them, successfully send logs to output, and close all harversters one-by-one when them reaching EOF. After that pods are restarting, and all repeat again.
But option "close_eof" is disable by default, and as you can see in my config below - i'm also specialy set close_eof: false.
My log, i'm replace some text in logpath with dots.
2021-04-09T05:06:57.984Z INFO log/harvester.go:331 End of file reached: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../0.log. Closing because close_eof is enabled.
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO log/harvester.go:329 Reader was closed: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../setup/0.log. Closing.
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO beater/crawler.go:178 Crawler stopped
2021-04-09T05:06:57.984Z INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2021-04-09T05:06:57.984Z INFO [registrar] registrar/registrar.go:166 Ending Registrar
2021-04-09T05:06:57.985Z INFO [registrar] registrar/registrar.go:137 Registrar stopped
2021-04-09T05:06:58.118Z INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":200000}},"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","stats":{"periods":124}},"cpuacct":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","total":{"ns":998256663}},"memory":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","mem":{"limit":{"bytes":524288000},"usage":{"bytes":99487744}}}},"cpu":{"system":{"ticks":140,"time":{"ms":141}},"total":{"ticks":700,"time":{"ms":701},"value":700},"user":{"ticks":560,"time":{"ms":560}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"fc7f60a2-148f-4aad-aef9-42b5281f81e2","uptime":{"ms":38664}},"memstats":{"gc_next":94545008,"memory_alloc":85914152,"memory_sys":145838088,"memory_total":234124024,"rss":139051008},"runtime":{"goroutines":30}},"filebeat":{"events":{"active":7,"added":1691,"done":1684},"harvester":{"closed":46,"open_files":0,"running":0,"started":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1505,"active":0,"batches":36,"total":1505},"type":"kafka"},"outputs":{"kafka":{"bytes_read":11150,"bytes_write":320209}},"pipeline":{"clients":0,"events":{"active":7,"filtered":179,"published":1512,"retry":899,"total":1691},"queue":{"acked":1505}}},"registrar":{"states":{"cleanup":1,"current":96,"update":1684},"writes":{"success":135,"total":135}},"system":{"cpu":{"cores":4},"load":{"1":1.38,"15":1.62,"5":1.34,"norm":{"1":0.345,"15":0.405,"5":0.335}}}}}}
2021-04-09T05:06:58.118Z INFO [monitoring] log/log.go:153 Uptime: 38.66501336s
2021-04-09T05:06:58.118Z INFO [monitoring] log/log.go:130 Stopping metrics logging.
2021-04-09T05:06:58.119Z INFO instance/beat.go:474 filebeat stopped.
My test config below, i'm trying set format container to cri, and change close options and timeouts. Any positive result get set scan_frequency to 1s - in that case filebeat reading more logs before closes in kind of EOF.
ignore_older: 240h
fields_under_root: true
filebeat.inputs:
- type: container
paths:
- "/var/log/pods/*/*/*.log"
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
close_eof: false
close_timeout: 10s
scan_frequency: 1s