# Filebeat reads a log file but loses some lines **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 24, 2018, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922 "2018-08-24T13:17:13Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Steve1](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Steve1](https://discuss.elastic.co/u/Steve1)\ **Post date:** [August 24, 2018, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922/1 "2018-08-24T13:17:14Z")
So i use filebeat 6.4.0. And i produce 1.000.000 lines with log4j2 to a main.log. Then i execute filebeat to read this main.log and writes the output to file. But filebeat dont have 1.000.000 lines but fewer. Why that? ## filebeat.yml filebeat.registry\_file: registry.eCommTest4 #=========================== Filebeat prospectors ============================= filebeat.prospectors: - type: log enabled: true paths: #======== File Output ========================================================= output.file: #Boolean flag to enable or disable the output module. enabled: true #Path to the directory where to save the generated files. The option is #mandatory. path: /home/steve/Desktop/FilebeatOutput/ #Name of the generated files. The default is `filebeat` and it generates filename: siebel.txt ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [August 24, 2018, 2:51pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922/2 "2018-08-24T14:51:35Z")
Hello @Steve1, This look like a strange behavior, I have a few questions. - How many events are not present? - If you diff the file are the missing events located in once place? - Did you start the tests multiple times with the same target file, if the file didn't not change `inode`, Filebeat would keep an offset or previous run in the `data/registry` file and will start reading from that and potentially miss events. Thanks ---
**Author:** ![Steve1](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Steve1](https://discuss.elastic.co/u/Steve1)\ **Post date:** [August 24, 2018, 5:00pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922/3 "2018-08-24T17:00:17Z")
1. From the 1.000.000 lines, filebeat output files (which is 7 files with 10MB size each), have from 79117 line until 1.000.000 line. That means that the line 0 until 79116 are missing from filebeat. 2. The events (actually are just lines with a simple txt and a counter) are located only in one file, called main.log. Then, filebeat produces 7 output files, with each file size almost 10MB. 3. Each time i reproduce the same process, i always delete the data/registry file. And each time, always delete the target file (main.log). ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [August 24, 2018, 5:40pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922/4 "2018-08-24T17:40:56Z")
@Steve1 This expected behavior, the file output rotates and replace files, by default Filebeat only keeps 7 files and will overwrite older files. > The maximum number of files to save under [`path`](https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html#path). When this number of files is reached, the oldest file is deleted, and the rest of the files are shifted from last to first. The number of files must be between 2 and 1024. The default is 7. If you look at [Configure the File output | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html#_literal_number_of_files_literal) Also note that that by default we rotate at every 10MB. > The maximum size in kilobytes of each file. When this size is reached, the files are rotated. The default value is 10240 KB. > **[Configure the File output | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html#_literal_rotate_every_kb_literal)** So if you want to test that filebeat persists all your events to disk you will need to play with the above options. ---
**Author:** ![Steve1](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Steve1](https://discuss.elastic.co/u/Steve1)\ **Post date:** [August 24, 2018, 7:35pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922/5 "2018-08-24T19:35:16Z")
Thank you. Honestly, just activating these 2 parameters, my problem got solved. Really thanks for the help. Just increased the number\_of\_files: 25 (from default 7), and worked! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 21, 2018, 7:46pm UTC](https://discuss.elastic.co/t/filebeat-reads-a-log-file-but-loses-some-lines/145922/6 "2018-09-21T19:46:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.