I am configuring filebeat in a kubernetes cluster.
Below is my config inside Pod:
cat /etc/filebeat.yml
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: ["ctst-log-app21.example.com:9200"]
username: 'filebeat'
password: 'filebeat'
index: 'cnv-k8s-%{+yyyy.MM.dd}'
setup.template:
name: 'cnv-k8s'
pattern: 'cnv-k8s-*'
enabled: false
# ILM setup
# ---------
setup.ilm:
enabled: true
rollover_alias: "cnv-k8s-%{+yyyy.MM.dd}"
policy_name: cnv-k8s
check_exists: false
overwrite: true
# Template setup
# --------------
setup.template:
name: 'cnv-k8s'
pattern: 'cnv-k8s-*'
enabled: true
If I look in /var/log/containers/ I see pod's log files.
Below you can see debug logs. Some "no such file or directory" erros here, but these files are in /var/log/containers/ inside pod.
What's wrong?
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input.(*Runner).Run","file.name":"input/input.go","file.line":137},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/cceaddon-npd-t9hn9_kube-system_npd-8c889212e1838ae0e9820a5fa6794868d6926ba5ce983efcc1b001127152146e.log) failed: stat /var/log/containers/cceaddon-npd-t9hn9_kube-system_npd-8c889212e1838ae0e9820a5fa6794868d6926ba5ce983efcc1b001127152146e.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/cceaddon-npd-t9hn9_kube-system_npd-bootstrap-f20f0d588d6ad3dfd75c60ced589b2a2b1b27cf1d7f8adf1d953a739380cdcaf.log) failed: stat /var/log/containers/cceaddon-npd-t9hn9_kube-system_npd-bootstrap-f20f0d588d6ad3dfd75c60ced589b2a2b1b27cf1d7f8adf1d953a739380cdcaf.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/cceaddon-npd-t9hn9_kube-system_npd-cc7beb422d8b41bde1006ed4d2cc8ce9dd144d898b21e03b706604115490c8cc.log) failed: stat /var/log/containers/cceaddon-npd-t9hn9_kube-system_npd-cc7beb422d8b41bde1006ed4d2cc8ce9dd144d898b21e03b706604115490c8cc.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/classic-ifl-tariffication-service-7b75bbd9c6-x8wrm_cnv-dev_istio-init-8e5c00421f588a9c8ded2d8d8cf311f9bd114b701e918978d04c7285ebceb7a0.log) failed: stat /var/log/containers/classic-ifl-tariffication-service-7b75bbd9c6-x8wrm_cnv-dev_istio-init-8e5c00421f588a9c8ded2d8d8cf311f9bd114b701e918978d04c7285ebceb7a0.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/classic-ifl-tariffication-service-7b75bbd9c6-x8wrm_cnv-dev_istio-proxy-905ffb002d8c7c526237d0591cf40e4d3baa972233667da4fc87cff4ceb4e171.log) failed: stat /var/log/containers/classic-ifl-tariffication-service-7b75bbd9c6-x8wrm_cnv-dev_istio-proxy-905ffb002d8c7c526237d0591cf40e4d3baa972233667da4fc87cff4ceb4e171.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/classic-ifl-tariffication-service-7b75bbd9c6-x8wrm_cnv-dev_service-244fadfbed03649fed16a55ec2ae22a3cecaa24b170fd48c52e6ab4b0e392feb.log) failed: stat /var/log/containers/classic-ifl-tariffication-service-7b75bbd9c6-x8wrm_cnv-dev_service-244fadfbed03649fed16a55ec2ae22a3cecaa24b170fd48c52e6ab4b0e392feb.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/cnv-kafka-connect-service-6559976755-s84fr_cnv-dev_istio-init-37e87bc237f6c49a8290721661b062b1971c93c6d5c81430c0b305fe64b65031.log) failed: stat /var/log/containers/cnv-kafka-connect-service-6559976755-s84fr_cnv-dev_istio-init-37e87bc237f6c49a8290721661b062b1971c93c6d5c81430c0b305fe64b65031.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/cnv-kafka-connect-service-6559976755-s84fr_cnv-dev_istio-proxy-4bf3b300701d261c2d77bcb5f96ceef2aa9e07be464b38c23b804211613913e4.log) failed: stat /var/log/containers/cnv-kafka-connect-service-6559976755-s84fr_cnv-dev_istio-proxy-4bf3b300701d261c2d77bcb5f96ceef2aa9e07be464b38c23b804211613913e4.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/cnv-kafka-connect-service-6559976755-s84fr_cnv-dev_service-74cffe0f2a1c4a67bbad4976ce061d2a71be29f6d7dcc6bdbfcc1f54739a231d.log) failed: stat /var/log/containers/cnv-kafka-connect-service-6559976755-s84fr_cnv-dev_service-74cffe0f2a1c4a67bbad4976ce061d2a71be29f6d7dcc6bdbfcc1f54739a231d.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/coredns-5744875d4f-bf2df_kube-system_coredns-7b972bd409234c182c1b63854fabc9778c9b4e31b75a1305e856057b3c76671c.log) failed: stat /var/log/containers/coredns-5744875d4f-bf2df_kube-system_coredns-7b972bd409234c182c1b63854fabc9778c9b4e31b75a1305e856057b3c76671c.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/everest-csi-controller-6cf5d6985f-46tlk_kube-system_everest-csi-controller-44ba9944eb68a678435ecc5f72e6d2613bc443db4e5cbbafc10f0ba98e066ed8.log) failed: stat /var/log/containers/everest-csi-controller-6cf5d6985f-46tlk_kube-system_everest-csi-controller-44ba9944eb68a678435ecc5f72e6d2613bc443db4e5cbbafc10f0ba98e066ed8.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/everest-csi-controller-6cf5d6985f-46tlk_kube-system_everest-csi-controller-init-8cd4e285eb4e7f963d6e56d7c52b03b57131162dfdf2f71f80afbdceda07f6e4.log) failed: stat /var/log/containers/everest-csi-controller-6cf5d6985f-46tlk_kube-system_everest-csi-controller-init-8cd4e285eb4e7f963d6e56d7c52b03b57131162dfdf2f71f80afbdceda07f6e4.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/everest-csi-driver-5cgzg_kube-system_everest-csi-driver-579b00058d5631f73de81b03696a6dd364c348f652300d1f09975345f914fff3.log) failed: stat /var/log/containers/everest-csi-driver-5cgzg_kube-system_everest-csi-driver-579b00058d5631f73de81b03696a6dd364c348f652300d1f09975345f914fff3.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/everest-csi-driver-5cgzg_kube-system_everest-csi-driver-f0a45878e0c6c0c5de7978c0e9fd2a185f7a2fe8edad05fbd559a0a4ea88a7c4.log) failed: stat /var/log/containers/everest-csi-driver-5cgzg_kube-system_everest-csi-driver-f0a45878e0c6c0c5de7978c0e9fd2a185f7a2fe8edad05fbd559a0a4ea88a7c4.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/everest-csi-driver-5cgzg_kube-system_everest-csi-driver-init-56e45ddb25f128f85032468197ffbb5747b0c2f9e94272884b287ace3d937912.log) failed: stat /var/log/containers/everest-csi-driver-5cgzg_kube-system_everest-csi-driver-init-56e45ddb25f128f85032468197ffbb5747b0c2f9e94272884b287ace3d937912.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/filebeat-c8rp6_cnv-dev_filebeat-aa8c7f75578324c2b37a21669a6738d86ea29c116454f0f145143f306d5c1176.log) failed: stat /var/log/containers/filebeat-c8rp6_cnv-dev_filebeat-aa8c7f75578324c2b37a21669a6738d86ea29c116454f0f145143f306d5c1176.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/icagent-7swzc_kube-system_icagent-55d8175750fe6db4356be00a2ed8e0e46c8943761ea9134f7ff92c0c0cfd3edc.log) failed: stat /var/log/containers/icagent-7swzc_kube-system_icagent-55d8175750fe6db4356be00a2ed8e0e46c8943761ea9134f7ff92c0c0cfd3edc.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/icagent-7swzc_kube-system_icagent-a369c1d693245a7a4393b79594167f42be7788d46f2a0e19a227a47540eaa286.log) failed: stat /var/log/containers/icagent-7swzc_kube-system_icagent-a369c1d693245a7a4393b79594167f42be7788d46f2a0e19a227a47540eaa286.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/node-local-dns-admission-controller-55b8c66dd4-m5mwf_kube-system_localdns-bootstrap-94e8aaf5666bf45f8a2680ff6871ed33abfd3f899b05257156f7b22b37f67307.log) failed: stat /var/log/containers/node-local-dns-admission-controller-55b8c66dd4-m5mwf_kube-system_localdns-bootstrap-94e8aaf5666bf45f8a2680ff6871ed33abfd3f899b05257156f7b22b37f67307.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/node-local-dns-admission-controller-55b8c66dd4-m5mwf_kube-system_webhook-0a04873a1a4e78330a4452f4d3e4ef770599fd3d8bec6b877121519020a13d97.log) failed: stat /var/log/containers/node-local-dns-admission-controller-55b8c66dd4-m5mwf_kube-system_webhook-0a04873a1a4e78330a4452f4d3e4ef770599fd3d8bec6b877121519020a13d97.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/node-local-dns-lgsff_kube-system_node-cache-0d3c49af5b45cd5a151ca787a71bf4a9415f849ac8bb4b97c614f66f50e5c9b2.log) failed: stat /var/log/containers/node-local-dns-lgsff_kube-system_node-cache-0d3c49af5b45cd5a151ca787a71bf4a9415f849ac8bb4b97c614f66f50e5c9b2.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/node-local-dns-lgsff_kube-system_node-cache-92d334248a1dc850eef85c11377e4665000566666ca01894b11f529fa2d57d25.log) failed: stat /var/log/containers/node-local-dns-lgsff_kube-system_node-cache-92d334248a1dc850eef85c11377e4665000566666ca01894b11f529fa2d57d25.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/node-problem-controller-85f94d9b9c-p7wpx_kube-system_npc-3b7d871bb70d65baa9523b5b4558f371058e62e6f2645fe0c3c64aae1ef3c85f.log) failed: stat /var/log/containers/node-problem-controller-85f94d9b9c-p7wpx_kube-system_npc-3b7d871bb70d65baa9523b5b4558f371058e62e6f2645fe0c3c64aae1ef3c85f.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/postgresql-0_cnv-dev_istio-init-2f708894156af2ab297c5179c34a2c60bc31b7f1523b2e6f7010ab7d1d004b83.log) failed: stat /var/log/containers/postgresql-0_cnv-dev_istio-init-2f708894156af2ab297c5179c34a2c60bc31b7f1523b2e6f7010ab7d1d004b83.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/postgresql-0_cnv-dev_istio-proxy-c5b71b50d14761b8dbc1fb953dacdc624ba659747d7162bc0038772fe7a94685.log) failed: stat /var/log/containers/postgresql-0_cnv-dev_istio-proxy-c5b71b50d14761b8dbc1fb953dacdc624ba659747d7162bc0038772fe7a94685.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/postgresql-0_cnv-dev_postgresql-1b7a0bf197ee0ec92fd5cb4e7224a1cc9fce031ad23de65313f9f8f99e30f7c5.log) failed: stat /var/log/containers/postgresql-0_cnv-dev_postgresql-1b7a0bf197ee0ec92fd5cb4e7224a1cc9fce031ad23de65313f9f8f99e30f7c5.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/quote-analysis-service-68fdcdf59f-nv82h_cnv-dev_istio-init-386d1cfffd4923246dc0d120b1f232a529d5e7f25a6f8c2ddf0f23b75200c5ba.log) failed: stat /var/log/containers/quote-analysis-service-68fdcdf59f-nv82h_cnv-dev_istio-init-386d1cfffd4923246dc0d120b1f232a529d5e7f25a6f8c2ddf0f23b75200c5ba.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/quote-analysis-service-68fdcdf59f-nv82h_cnv-dev_istio-proxy-2b2ddf6114ca08ecd5d0fcfec848f204e196843811e3d3408738e753e2dab0d0.log) failed: stat /var/log/containers/quote-analysis-service-68fdcdf59f-nv82h_cnv-dev_istio-proxy-2b2ddf6114ca08ecd5d0fcfec848f204e196843811e3d3408738e753e2dab0d0.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/quote-analysis-service-68fdcdf59f-nv82h_cnv-dev_service-1a32f90b2be28dc0668330101317408f20a1126e7635afc18ac7b36a34c1dadf.log) failed: stat /var/log/containers/quote-analysis-service-68fdcdf59f-nv82h_cnv-dev_service-1a32f90b2be28dc0668330101317408f20a1126e7635afc18ac7b36a34c1dadf.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/quote-log-service-577d856568-pffhp_cnv-dev_istio-init-9113995bc127467f31d1469d93322fc3fda86ac40a8941737bb7c7ba28522144.log) failed: stat /var/log/containers/quote-log-service-577d856568-pffhp_cnv-dev_istio-init-9113995bc127467f31d1469d93322fc3fda86ac40a8941737bb7c7ba28522144.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/quote-log-service-577d856568-pffhp_cnv-dev_istio-proxy-54f794a34c565a9eb842415c485b09465408186ffbc20d37d168009fd21ba392.log) failed: stat /var/log/containers/quote-log-service-577d856568-pffhp_cnv-dev_istio-proxy-54f794a34c565a9eb842415c485b09465408186ffbc20d37d168009fd21ba392.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/quote-log-service-577d856568-pffhp_cnv-dev_service-5d7be189c913f58ac2bc838c4ec35fa79f376b6e5c899372c930604f71cb57bf.log) failed: stat /var/log/containers/quote-log-service-577d856568-pffhp_cnv-dev_service-5d7be189c913f58ac2bc838c4ec35fa79f376b6e5c899372c930604f71cb57bf.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/redis-master-0_cnv-dev_istio-init-c15a5fcd04a79b2ab4207bcd4cd4073db26d81eb25110a7c61ea3f1dc3b20692.log) failed: stat /var/log/containers/redis-master-0_cnv-dev_istio-init-c15a5fcd04a79b2ab4207bcd4cd4073db26d81eb25110a7c61ea3f1dc3b20692.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/redis-master-0_cnv-dev_istio-proxy-6241c0f8a87a9482aec0e3a790a863bc73427b000ef04d91817e043ee48dee24.log) failed: stat /var/log/containers/redis-master-0_cnv-dev_istio-proxy-6241c0f8a87a9482aec0e3a790a863bc73427b000ef04d91817e043ee48dee24.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/redis-master-0_cnv-dev_redis-02505cae82b19e8a7ffbd029035884db1c59b248818ad6eada6934ea5304dc9d.log) failed: stat /var/log/containers/redis-master-0_cnv-dev_redis-02505cae82b19e8a7ffbd029035884db1c59b248818ad6eada6934ea5304dc9d.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/risk-monitoring-service-665755dc86-77hvt_cnv-dev_istio-init-747a3812bf5dfd43da377660c0e0245443c5ccba31c9cd1a1e1229d2ecef43c2.log) failed: stat /var/log/containers/risk-monitoring-service-665755dc86-77hvt_cnv-dev_istio-init-747a3812bf5dfd43da377660c0e0245443c5ccba31c9cd1a1e1229d2ecef43c2.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/risk-monitoring-service-665755dc86-77hvt_cnv-dev_istio-proxy-1e2fa19192c492e59bc6c01e37195e8b0e9bbaadcd27fc5f4250a067d7caa8d2.log) failed: stat /var/log/containers/risk-monitoring-service-665755dc86-77hvt_cnv-dev_istio-proxy-1e2fa19192c492e59bc6c01e37195e8b0e9bbaadcd27fc5f4250a067d7caa8d2.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/risk-monitoring-service-665755dc86-77hvt_cnv-dev_service-ddb528ac6f6058f12124ae1ccf0f950ef6ddb03e384eb9e7022444b72d04fff7.log) failed: stat /var/log/containers/risk-monitoring-service-665755dc86-77hvt_cnv-dev_service-ddb528ac6f6058f12124ae1ccf0f950ef6ddb03e384eb9e7022444b72d04fff7.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/subject-decision-engine-service-7cdfbdb57f-skf96_cnv-dev_istio-init-59430c67186b3b0071468e87140bb8ee73d19dc676a786308f3f8cbaf65ec63f.log) failed: stat /var/log/containers/subject-decision-engine-service-7cdfbdb57f-skf96_cnv-dev_istio-init-59430c67186b3b0071468e87140bb8ee73d19dc676a786308f3f8cbaf65ec63f.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/subject-decision-engine-service-7cdfbdb57f-skf96_cnv-dev_istio-proxy-fd0e8e3aa89fcb2cabc292eae1f69c4fb4140cdeb9c5283dc370cf763658e61c.log) failed: stat /var/log/containers/subject-decision-engine-service-7cdfbdb57f-skf96_cnv-dev_istio-proxy-fd0e8e3aa89fcb2cabc292eae1f69c4fb4140cdeb9c5283dc370cf763658e61c.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/subject-decision-engine-service-7cdfbdb57f-skf96_cnv-dev_service-4eaebea86f306fda5599ec267242a1a54843d82c20963f22dcadd6391e58622c.log) failed: stat /var/log/containers/subject-decision-engine-service-7cdfbdb57f-skf96_cnv-dev_service-4eaebea86f306fda5599ec267242a1a54843d82c20963f22dcadd6391e58622c.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/swiss-army-knife_cnv-dev_istio-init-33c75915ba45b26035fde5064b84ecdfe40dcb1b1602721d1e98bfbe846d3653.log) failed: stat /var/log/containers/swiss-army-knife_cnv-dev_istio-init-33c75915ba45b26035fde5064b84ecdfe40dcb1b1602721d1e98bfbe846d3653.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/swiss-army-knife_cnv-dev_istio-proxy-759038c2959fe677370101234cd418bfabc5b450a4fcc2cb822d464c27c9d4a2.log) failed: stat /var/log/containers/swiss-army-knife_cnv-dev_istio-proxy-759038c2959fe677370101234cd418bfabc5b450a4fcc2cb822d464c27c9d4a2.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).getFiles","file.name":"log/input.go","file.line":354},"message":"stat(/var/log/containers/swiss-army-knife_cnv-dev_swiss-army-knife-087db3efdd4db4fa335629a0dd9ff8935192b2eb50b980e4e31d868afc81300c.log) failed: stat /var/log/containers/swiss-army-knife_cnv-dev_swiss-army-knife-087db3efdd4db4fa335629a0dd9ff8935192b2eb50b980e4e31d868afc81300c.log: no such file or directory","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-01-31T12:56:11.239Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"3076ea10-74d2-4722-9f5b-dfb09eea57ad","ecs.version":"1.6.0"}