Доброго дня! Проблема такая - файлбит (релизы 7.12 и 7.11.2) разворачиваю демонсетом в кластер k8s openshift, рантайм cri, логи контейнеров забираются из примонтированной директории, после запуска с десяток секунд всё ок, харвестры успешно читают логи, затем начинают натыкаться на EOF и файлбит отключается, затем перезапускается дейплойментом и круг замыкается.
Лог, почему то отрабатываются закрытия харвестров при EOF, хотя close_eof по дефолту отключено, и я уже специально указал close_eof: false явно в конфиге
2021-04-09T05:06:57.984Z INFO log/harvester.go:331 End of file reached: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../0.log. Closing because close_eof is enabled.
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO log/harvester.go:329 Reader was closed: /var/log/pods/openshift-kube-apiserver_kube-apiserver-.........../setup/0.log. Closing.
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO harvester/forwarder.go:52 Input outlet closed
2021-04-09T05:06:57.984Z INFO beater/crawler.go:178 Crawler stopped
2021-04-09T05:06:57.984Z INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2021-04-09T05:06:57.984Z INFO [registrar] registrar/registrar.go:166 Ending Registrar
2021-04-09T05:06:57.985Z INFO [registrar] registrar/registrar.go:137 Registrar stopped
2021-04-09T05:06:58.118Z INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":200000}},"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","stats":{"periods":124}},"cpuacct":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","total":{"ns":998256663}},"memory":{"id":"crio-9fe1c3e817e76afe2382d4d648f55cc6424f592b48cfb4ab31eaad42ec1f64c8.scope","mem":{"limit":{"bytes":524288000},"usage":{"bytes":99487744}}}},"cpu":{"system":{"ticks":140,"time":{"ms":141}},"total":{"ticks":700,"time":{"ms":701},"value":700},"user":{"ticks":560,"time":{"ms":560}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"fc7f60a2-148f-4aad-aef9-42b5281f81e2","uptime":{"ms":38664}},"memstats":{"gc_next":94545008,"memory_alloc":85914152,"memory_sys":145838088,"memory_total":234124024,"rss":139051008},"runtime":{"goroutines":30}},"filebeat":{"events":{"active":7,"added":1691,"done":1684},"harvester":{"closed":46,"open_files":0,"running":0,"started":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1505,"active":0,"batches":36,"total":1505},"type":"kafka"},"outputs":{"kafka":{"bytes_read":11150,"bytes_write":320209}},"pipeline":{"clients":0,"events":{"active":7,"filtered":179,"published":1512,"retry":899,"total":1691},"queue":{"acked":1505}}},"registrar":{"states":{"cleanup":1,"current":96,"update":1684},"writes":{"success":135,"total":135}},"system":{"cpu":{"cores":4},"load":{"1":1.38,"15":1.62,"5":1.34,"norm":{"1":0.345,"15":0.405,"5":0.335}}}}}}
2021-04-09T05:06:58.118Z INFO [monitoring] log/log.go:153 Uptime: 38.66501336s
2021-04-09T05:06:58.118Z INFO [monitoring] log/log.go:130 Stopping metrics logging.
2021-04-09T05:06:58.119Z INFO instance/beat.go:474 filebeat stopped.
Тестовый конфиг ниже, немного помогло увеличить частоту сканов, при этом файлбит вычитывает чуть больше логов прежде чем успевает закрыть все харвестры, close_eof: false не влияет, явно указание формата на cri тоже не повлияло (в этом конфиге нет, но проверено)
ignore_older: 240h
fields_under_root: true
filebeat.inputs:
- type: container
paths:
- "/var/log/pods/*/*/*.log"
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
close_eof: false
close_timeout: 10s
scan_frequency: 1s