Hi.
I use filebeat 7.6.1 and k8s autodiscovery that works good for each pod.
Example:
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
or:
- equals:
kubernetes.namespace: cis
- equals:
kubernetes.namespace: cicada
But filebeat doesn't collect the output from k8s cronjob. I configured the cronjob which prints some python output but this output isn't appeared in kibana.
I found similar issue - Filebeat isn`t collecting logs of short living containers like cronjobs
But there is no answer.
I added sleep command - still no output in kibana.
The logs in filebeat:
2020-12-07T13:51:05.724Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":75790,"time":{"ms":44}},"total":{"ticks":419890,"time":{"ms":241},"value":419890},"user":{"ticks":344100,"time":{"ms":197}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":29},"info":{"ephemeral_id":"7d1734fd-06bb-4314-a5cb-9fb7880e336c","uptime":{"ms":10470473}},"memstats":{"gc_next":24610864,"memory_alloc":20488048,"memory_total":37707607936},"runtime":{"goroutines":318}},"filebeat":{"events":{"active":16,"added":96,"done":80},"harvester":{"files":{"095608ef-f735-496a-ad7d-05e9a72a8e37":{"last_event_published_time":"2020-12-07T13:50:56.036Z","last_event_timestamp":"2020-12-07T13:50:50.220Z","read_offset":1811,"size":1810},"0dc3b52e-02aa-49c9-8447-ed10a495d8fe":{"last_event_published_time":"2020-12-07T13:51:04.874Z","last_event_timestamp":"2020-12-07T13:51:04.874Z","read_offset":9877,"size":4495},"176c875b-5005-486e-866c-4d8d3bc050b4":{"last_event_published_time":"","last_event_timestamp":"","name":"/var/log/containers/kubectl-1607349060-ktjvt_kube-logging_kubectl-f1401fc5f8e0a88c3d6b20f620fff4391ddb51644db9321cf0188eae829a0285.log","start_time":"2020-12-07T13:51:05.252Z"},"1bab1064-a140-4730-a765-9eb50f8b4444":{"last_event_published_time":"2020-12-07T13:51:02.800Z","last_event_timestamp":"2020-12-07T13:50:51.290Z","read_offset":2244,"size":2243},"224851ff-270d-4d15-973c-3b5341dfda4e":{"last_event_published_time":"2020-12-07T13:51:01.760Z","last_event_timestamp":"2020-12-07T13:50:59.876Z","read_offset":2253,"size":2253},"36c2f29a-b92c-412a-adfd-c8618591f158":{"last_event_published_time":"2020-12-07T13:50:57.375Z","last_event_timestamp":"2020-12-07T13:50:52.202Z","read_offset":2778,"size":3673},"5151a951-0e39-45c4-9e55-64128ed40bbc":{"last_event_published_time":"2020-12-07T13:51:03.800Z","last_event_timestamp":"2020-12-07T13:51:01.074Z","read_offset":667},"652aadb7-57d1-4d39-ab83-1c3b9580b332":{"last_event_published_time":"2020-12-07T13:50:46.527Z","last_event_timestamp":"2020-12-07T13:50:40.734Z","read_offset":3170,"size":6093},"65780a51-40ac-47c7-9024-f2626749ff03":{"last_event_published_time":"2020-12-07T13:50:50.510Z","last_event_timestamp":"2020-12-07T13:50:35.724Z","read_offset":3887,"size":3887},"77807dde-11ce-4c05-bf4d-b844abd103a8":{"last_event_published_time":"2020-12-07T13:51:04.815Z","last_event_timestamp":"2020-12-07T13:50:52.875Z","read_offset":4160,"size":3493},"9f0764e8-a745-4754-a691-cf70de9adcf5":{"size":930},"a15d15bd-9581-40c4-91f6-c461d093c56e":{"last_event_published_time":"","last_event_timestamp":"","name":"/var/log/containers/cicada-db-1607349060-k6d8v_cicada_cicada-db-ab6ddf4b67b42f49fe3bceff64ea5899bdca50c31a5350f45650f75216c6e081.log","start_time":"2020-12-07T13:51:05.223Z"},"ad82c393-a0d6-4f9a-b5df-07524f601e33":{"last_event_published_time":"2020-12-07T13:50:53.382Z","last_event_timestamp":"2020-12-07T13:50:42.111Z","read_offset":5841,"size":2923},"c6d1ee72-fd4c-4f8a-8e86-efc39f44490d":{"last_event_published_time":"2020-12-07T13:50:58.618Z","last_event_timestamp":"2020-12-07T13:50:42.925Z","read_offset":3062,"size":3062},"c77a08f6-5ef3-4575-8583-4d19ce605645":{"last_event_published_time":"2020-12-07T13:51:03.444Z","last_event_timestamp":"2020-12-07T13:51:00.651Z","read_offset":4349,"size":4348},"f41f8126-fdf8-4621-a40b-2e75e67959ab":{"last_event_published_time":"","last_event_timestamp":"","name":"/var/log/containers/cicada-uiapi-1607349060-tx74b_cicada_cicada-uiapi-a668c5ce1e7e40940b7c374726730fec0c6b37b80799d6215b4a408cca2feca8.log","size":72,"start_time":"2020-12-07T13:51:05.236Z"}},"open_files":17,"running":17,"started":3}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":77,"batches":18,"total":77},"read":{"bytes":6739},"write":{"bytes":119654}},"pipeline":{"clients":28,"events":{"active":16,"filtered":3,"published":93,"total":96},"queue":{"acked":77}}},"registrar":{"states":{"current":504,"update":80},"writes":{"success":21,"total":21}},"system":{"load":{"1":2.25,"15":2.48,"5":2.78,"norm":{"1":0.5625,"15":0.62,"5":0.695}}}}}}
2020-12-07T13:52:05.989Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349120-5t2np_cicada_cicada-uiapi-78192c420e0900b49d854f393810db97c3ffabeb6b3e8a28c7ca3d62cd36a828.log
2020-12-07T13:53:05.768Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349180-wzg9h_cicada_cicada-uiapi-75d4c468d8daf60210bfc8e50c95ed532b4a9c5771870a1c9a017c59c7af046b.log
2020-12-07T13:54:08.310Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349240-848nk_cicada_cicada-uiapi-4fe3f1c460ac5756034b44e35dd212601fa742bb46b2977cd7b724c840ab224c.log
2020-12-07T13:55:07.795Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349300-hpr72_cicada_cicada-uiapi-20409882c16bf648d118918e1340c56424edd8f587c1efbfdbee6207ab098dff.log
2020-12-07T13:56:08.722Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349360-qntnp_cicada_cicada-uiapi-47ddc90a8db1d8dad2d003a0fcfbbc90e2a15f5775d846b341a05fb4c5ae75e6.log
2020-12-07T13:57:09.705Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349420-rjl46_cicada_cicada-uiapi-473fd934fc8cf1bb2b6b984a383ba2124a000f0fefb265957cdcc1c4d913ceb2.log
2020-12-07T13:58:10.098Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349480-cgkff_cicada_cicada-uiapi-746ec4eafe414cdda3e69ce19f0cf6678373976a5ef1fdc583aa6d5c55d3439f.log
2020-12-07T13:59:10.493Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349540-sp8tn_cicada_cicada-uiapi-78e4055bd8cd71e5a08d8fcd01889ab6fa70f2a504e7586482423d843b369a6c.log
2020-12-07T14:00:11.296Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349600-4ttwt_cicada_cicada-uiapi-1c7918ac5afaa4af12b6d95435d780baefc830e382fab2c80ca55b0f1811831f.log
2020-12-07T14:01:12.079Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349660-r257b_cicada_cicada-uiapi-d121c741226657f1d239c2f5172cd6287f9048e2c608187497388b2246c61c56.log
2020-12-07T14:02:12.774Z INFO log/harvester.go:297 Harvester started for file: /var/log/containers/cicada-uiapi-1607349720-zdgp5_cicada_cicada-uiapi-f76a3429ce20fc2e7b794901a29d8a00870a351ef24702fa1b0fc03efd807002.log
As you see the harvester is started for cicada-uiapi-* pod but no output in kibana 
UPD:
sometimes the cronjob output are logged in elasticsearch but not for each pod of cronjob. I have cronjob that runs pods each minute. For example, it launched 10 pods. And may be only from the 4th pod we will get the output to elasticsearch.
Please, help.