Filebeat stopped after reaching EOF, with close_eof:false

Hi guys. Need help in that case. Have some trouble with deploying daemonset of filebeat's (releases 7.12 or 7.11.2) in openshift cluster by runtime cri. After starting container, filebeat open harvesters for finded files, read them, successfully send logs to output, and close all harversters one-by-one when them reaching EOF. After that pods are restarting, and all repeat again.
But option "close_eof" is disable by default, and as you can see in my config below - i'm also specialy set close_eof: false.

My log, i'm replace some text in logpath with dots.

2021-04-09T05:06:57.984Z	INFO	log/harvester.go:331	End of file reached: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../0.log. Closing because close_eof is enabled.
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	log/harvester.go:329	Reader was closed: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../setup/0.log. Closing.
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	beater/crawler.go:178	Crawler stopped
2021-04-09T05:06:57.984Z	INFO	[registrar]	registrar/registrar.go:132	Stopping Registrar
2021-04-09T05:06:57.984Z	INFO	[registrar]	registrar/registrar.go:166	Ending Registrar
2021-04-09T05:06:57.985Z	INFO	[registrar]	registrar/registrar.go:137	Registrar stopped
2021-04-09T05:06:58.118Z	INFO	[monitoring]	log/log.go:152	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":200000}},"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","stats":{"periods":124}},"cpuacct":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","total":{"ns":998256663}},"memory":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","mem":{"limit":{"bytes":524288000},"usage":{"bytes":99487744}}}},"cpu":{"system":{"ticks":140,"time":{"ms":141}},"total":{"ticks":700,"time":{"ms":701},"value":700},"user":{"ticks":560,"time":{"ms":560}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"fc7f60a2-148f-4aad-aef9-42b5281f81e2","uptime":{"ms":38664}},"memstats":{"gc_next":94545008,"memory_alloc":85914152,"memory_sys":145838088,"memory_total":234124024,"rss":139051008},"runtime":{"goroutines":30}},"filebeat":{"events":{"active":7,"added":1691,"done":1684},"harvester":{"closed":46,"open_files":0,"running":0,"started":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1505,"active":0,"batches":36,"total":1505},"type":"kafka"},"outputs":{"kafka":{"bytes_read":11150,"bytes_write":320209}},"pipeline":{"clients":0,"events":{"active":7,"filtered":179,"published":1512,"retry":899,"total":1691},"queue":{"acked":1505}}},"registrar":{"states":{"cleanup":1,"current":96,"update":1684},"writes":{"success":135,"total":135}},"system":{"cpu":{"cores":4},"load":{"1":1.38,"15":1.62,"5":1.34,"norm":{"1":0.345,"15":0.405,"5":0.335}}}}}}
2021-04-09T05:06:58.118Z	INFO	[monitoring]	log/log.go:153	Uptime: 38.66501336s
2021-04-09T05:06:58.118Z	INFO	[monitoring]	log/log.go:130	Stopping metrics logging.
2021-04-09T05:06:58.119Z	INFO	instance/beat.go:474	filebeat stopped.

My test config below, i'm trying set format container to cri, and change close options and timeouts. Any positive result get set scan_frequency to 1s - in that case filebeat reading more logs before closes in kind of EOF.

    ignore_older: 240h
    fields_under_root: true
    filebeat.inputs:
    - type: container
      paths:
        - "/var/log/pods/*/*/*.log"
      multiline.pattern: '^[[:space:]]'
      multiline.negate: false
      multiline.match: after
      close_eof: false
      close_timeout: 10s
      scan_frequency: 1s

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.