Filebeat stopped после eof, при close_eof: false

Доброго дня! Проблема такая - файлбит (релизы 7.12 и 7.11.2) разворачиваю демонсетом в кластер k8s openshift, рантайм cri, логи контейнеров забираются из примонтированной директории, после запуска с десяток секунд всё ок, харвестры успешно читают логи, затем начинают натыкаться на EOF и файлбит отключается, затем перезапускается дейплойментом и круг замыкается.

Лог, почему то отрабатываются закрытия харвестров при EOF, хотя close_eof по дефолту отключено, и я уже специально указал close_eof: false явно в конфиге

2021-04-09T05:06:57.984Z	INFO	log/harvester.go:331	End of file reached: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../0.log. Closing because close_eof is enabled.
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	log/harvester.go:329	Reader was closed: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../setup/0.log. Closing.
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	harvester/forwarder.go:52	Input outlet closed
2021-04-09T05:06:57.984Z	INFO	beater/crawler.go:178	Crawler stopped
2021-04-09T05:06:57.984Z	INFO	[registrar]	registrar/registrar.go:132	Stopping Registrar
2021-04-09T05:06:57.984Z	INFO	[registrar]	registrar/registrar.go:166	Ending Registrar
2021-04-09T05:06:57.985Z	INFO	[registrar]	registrar/registrar.go:137	Registrar stopped
2021-04-09T05:06:58.118Z	INFO	[monitoring]	log/log.go:152	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":200000}},"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","stats":{"periods":124}},"cpuacct":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","total":{"ns":998256663}},"memory":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","mem":{"limit":{"bytes":524288000},"usage":{"bytes":99487744}}}},"cpu":{"system":{"ticks":140,"time":{"ms":141}},"total":{"ticks":700,"time":{"ms":701},"value":700},"user":{"ticks":560,"time":{"ms":560}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"fc7f60a2-148f-4aad-aef9-42b5281f81e2","uptime":{"ms":38664}},"memstats":{"gc_next":94545008,"memory_alloc":85914152,"memory_sys":145838088,"memory_total":234124024,"rss":139051008},"runtime":{"goroutines":30}},"filebeat":{"events":{"active":7,"added":1691,"done":1684},"harvester":{"closed":46,"open_files":0,"running":0,"started":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1505,"active":0,"batches":36,"total":1505},"type":"kafka"},"outputs":{"kafka":{"bytes_read":11150,"bytes_write":320209}},"pipeline":{"clients":0,"events":{"active":7,"filtered":179,"published":1512,"retry":899,"total":1691},"queue":{"acked":1505}}},"registrar":{"states":{"cleanup":1,"current":96,"update":1684},"writes":{"success":135,"total":135}},"system":{"cpu":{"cores":4},"load":{"1":1.38,"15":1.62,"5":1.34,"norm":{"1":0.345,"15":0.405,"5":0.335}}}}}}
2021-04-09T05:06:58.118Z	INFO	[monitoring]	log/log.go:153	Uptime: 38.66501336s
2021-04-09T05:06:58.118Z	INFO	[monitoring]	log/log.go:130	Stopping metrics logging.
2021-04-09T05:06:58.119Z	INFO	instance/beat.go:474	filebeat stopped.

Тестовый конфиг ниже, немного помогло увеличить частоту сканов, при этом файлбит вычитывает чуть больше логов прежде чем успевает закрыть все харвестры, close_eof: false не влияет, явно указание формата на cri тоже не повлияло (в этом конфиге нет, но проверено)

    ignore_older: 240h
    fields_under_root: true
    filebeat.inputs:
    - type: container
      paths:
        - "/var/log/pods/*/*/*.log"
      multiline.pattern: '^[[:space:]]'
      multiline.negate: false
      multiline.match: after
      close_eof: false
      close_timeout: 10s
      scan_frequency: 1s

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.