# Filebeat cannot collect the first few log lines

**URL:** <https://discuss.elastic.co/t/filebeat-cannot-collect-the-first-few-log-lines/369988>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 4, 2024, 8:33am UTC](https://discuss.elastic.co/t/filebeat-cannot-collect-the-first-few-log-lines/369988 "2024-11-04T08:33:16Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhangli178](https://avatars.discourse-cdn.com/v4/letter/z/bbe5ce/32.png) [@zhangli178](https://discuss.elastic.co/u/zhangli178)\
**Post date:** [November 4, 2024, 8:33am UTC](https://discuss.elastic.co/t/filebeat-cannot-collect-the-first-few-log-lines/369988/1 "2024-11-04T08:33:16Z")

</div>

I have a script to create a batch of pods to run on my k8s cluster. On each node, ECK beat deamon of type filebeat is running, they collect the logs from paths /var/log/containers/\*.log and send to logstash for further processing. Each pod only has one container and it will print environment variables to stdout of container at the beginning. When I check the collected logs through Kibana, I cannot find env var logs from a few pod, but env var logs do generate because I can see them using kubectl logs. I do not know why, anyone can explain it? Thanks.

---

<div class="post-metadata">

**Author:** ![zhangli178](https://avatars.discourse-cdn.com/v4/letter/z/bbe5ce/32.png) [@zhangli178](https://discuss.elastic.co/u/zhangli178)\
**Post date:** [November 4, 2024, 8:44am UTC](https://discuss.elastic.co/t/filebeat-cannot-collect-the-first-few-log-lines/369988/2 "2024-11-04T08:44:45Z")

</div>

From debug logs of beat, it seems like filebeat recognizes the new container log file as the existing file it has ever handled. The related logs are as follows.

```auto
{"log.level":"debug","@timestamp":"2024-11-04T06:43:51.518Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.getFileState","file.name":"log/input.go","file.line":476},"message":"Check file for harvesting: /var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","service.name":"filebeat","input_id":"646e359f-d58c-4e19-bd61-5686817ee653","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-04T06:43:51.518Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":574},"message":"Update existing file for harvesting: /var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log, offset: 5018","service.name":"filebeat","input_id":"646e359f-d58c-4e19-bd61-5686817ee653","source_file":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","state_id":"native::4869343-64768","finished":false,"os_id":"4869343-64768","old_source":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","old_finished":false,"old_os_id":"4869343-64768","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-04T06:43:51.518Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":627},"message":"Harvester for file is still running: /var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","service.name":"filebeat","input_id":"646e359f-d58c-4e19-bd61-5686817ee653","source_file":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","state_id":"native::4869343-64768","finished":false,"os_id":"4869343-64768","old_source":"/var/log/containers/test-job-env-70-pod-0_zhang-li178_runtime-3d716f5d2075ff6c2860f614be4154887b9f1d8b7ad8da61b611d8138b1a53b0.log","old_finished":false,"old_os_id":"4869343-64768","ecs.version":"1.6.0"}

```