Filebeat cannot collect the first few log lines

Elastic Stack Beats
1

I have a script to create a batch of pods to run on my k8s cluster. On each node, ECK beat deamon of type filebeat is running, they collect the logs from paths /var/log/containers/*.log and send to logstash for further processing. Each pod only has one container and it will print environment variables to stdout of container at the beginning. When I check the collected logs through Kibana, I cannot find env var logs from a few pod, but env var logs do generate because I can see them using kubectl logs. I do not know why, anyone can explain it? Thanks.

2

From debug logs of beat, it seems like filebeat recognizes the new container log file as the existing file it has ever handled. The related logs are as follows.

{"log.level":"debug","@timestamp":"2024-11-04T06:43:51.518Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.getFileState","file.name":"log/input.go","file.line":476},"message":"Check file for harvesting: /var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","service.name":"filebeat","input_id":"646e359f-d58c-4e19-bd61-5686817ee653","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-04T06:43:51.518Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":574},"message":"Update existing file for harvesting: /var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log, offset: 5018","service.name":"filebeat","input_id":"646e359f-d58c-4e19-bd61-5686817ee653","source_file":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","state_id":"native::4869343-64768","finished":false,"os_id":"4869343-64768","old_source":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","old_finished":false,"old_os_id":"4869343-64768","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-04T06:43:51.518Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":627},"message":"Harvester for file is still running: /var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","service.name":"filebeat","input_id":"646e359f-d58c-4e19-bd61-5686817ee653","source_file":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","state_id":"native::4869343-64768","finished":false,"os_id":"4869343-64768","old_source":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","old_finished":false,"old_os_id":"4869343-64768","ecs.version":"1.6.0"}