# Filebeats, unable to read all the data from log file **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 7, 2022, 5:54pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102 "2022-10-07T17:54:56Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [October 7, 2022, 5:54pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/1 "2022-10-07T17:54:56Z")
Hello Team, Greetings, I am trying to ingest data from log file to elastic via logstash. Here is the pipeline -\> LOG\_FILE \> FILEBEAT \> LOGSTASH \> ELASTIC. Not sure but recently been observing the logs are missing in elastic, upon checking I realized the log ends up in log file but not in the filebeat. I tested with ```auto output.file: path: "/tmp/filebeat" filename: filebeat ``` I didn't find any log in the file not sure why filebeat input is not pulling the logs. Here is my filebeat inputs ```auto filebeat.inputs: - type: log enabled: true paths: - '/var/www/myapp.log' close_renamed: true tail_files: true processors: - add_tags: tags: [myapp] target: "application" logging.to_files: true logging.level: debug logging.files: path: /var/log/filebeat name: filebeat rotateeverybytes: 104857600 keepfiles: 7 output.logstash: hosts: ["XX1:5044", "XX6:5044"] loadbalance: true ``` ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 10, 2022, 3:37pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/2 "2022-10-10T15:37:32Z")
Hey @adityak248, Is there any reason to use `close_renamed` in your case? If the files filebeat is reading are rotated by renaming them, they will be closed before being completely read, take a look to the [docs](https://www.elastic.co/guide/en/beats/filebeat/8.4/filebeat-input-log.html#filebeat-input-log-close-renamed) of this option. Same thing for `tail_files`. This option will make filebeat to start by the end of a file when opening it. In combination, both options may make Filebeat to stop reading files too soon, and to ignore the first lines of new files. I would suggest to try without these options, unless there is some strong reason to use them. Other things to consider: - Is there any reason to use Logstash in your deployment? You can send events directly from Filebeat to Elasticsearch and this would simplify your deployment. - Consider using the [`filestream` input](https://www.elastic.co/guide/en/beats/filebeat/8.4/filebeat-input-filestream.html), that is intended to replace the `log` input and solves some issues it had. ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [October 10, 2022, 5:25pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/3 "2022-10-10T17:25:08Z")
Hello @jsoriano Thanks for reaching back, ` close_renamed` I use this because the file gets renamed and moved away while a new file is created with same name as updated in the filebeat.yml. I have seen few cases where Filebeat keeps looking for the old file so using this option. `tail_file` : I did try removing this option but ended up with same result. I am deploying Filestream right now and will keep you posted with new changes. ```auto filebeat.inputs: - type: filestream paths: - '/var/www/myapp.log' ``` ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [October 10, 2022, 5:41pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/4 "2022-10-10T17:41:17Z")
Hello @jsoriano That didn't help me. It's the same. I tried to read the log with stdout on filebeat and nothing showed up. Can you please help me debug this? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 10, 2022, 7:02pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/5 "2022-10-10T19:02:46Z")
The problem is that no logs at all are collected, or only some log lines are lost? ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [October 10, 2022, 10:27pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/6 "2022-10-10T22:27:27Z")
Only some logs are not collected. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 11, 2022, 10:09am UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/7 "2022-10-11T10:09:02Z")
Are you sure that the rotations happen by moving the file and creating a new one? Or it is copying and truncating? Please take a look to these troubleshooting docs in case they give you some idea [Log rotation results in lost or duplicate events | Filebeat Reference [8.4] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/8.4/file-log-rotation.html) ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [October 12, 2022, 3:55pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/8 "2022-10-12T15:55:21Z")
Hello Jamie, The attached link is not helping me and here is my observation. I am seeing logs from account off hours but when I do try in peak hours I see no logs. This concludes that filebeat is not able to read all the logs. Any inputs on boosting filebeat performance? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 13, 2022, 6:17pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/9 "2022-10-13T18:17:30Z")
So is log rotation happening by copy and truncating the files, or by moving the file and creating a new one? Have you considered the idea of removing Logstash from the equation? Is there any reason you need it in your deployment? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 10, 2022, 8:17pm UTC](https://discuss.elastic.co/t/filebeats-unable-to-read-all-the-data-from-log-file/316102/10 "2022-11-10T20:17:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.