Filebeat reads a log file but loses some lines

Elastic Stack Beats
1

So i use filebeat 6.4.0. And i produce 1.000.000 lines with log4j2 to a main.log. Then i execute filebeat to read this main.log and writes the output to file. But filebeat dont have 1.000.000 lines but fewer. Why that?

filebeat.yml

filebeat.registry_file: registry.eCommTest4
#=========================== Filebeat prospectors =============================
filebeat.prospectors:

  • type: log
    enabled: true
    paths:

    • /home/steve/Desktop/LogFiles/main.log

    #close_inactive: 5m
    #close_timeout: 1h
    #ignore_older: 90m

#======== File Output =========================================================
output.file:
#Boolean flag to enable or disable the output module.
enabled: true

#Path to the directory where to save the generated files. The option is
#mandatory.
path: /home/steve/Desktop/FilebeatOutput/

#Name of the generated files. The default is filebeat and it generates
filename: siebel.txt

2

Hello @Steve1,

This look like a strange behavior, I have a few questions.

  • How many events are not present?
  • If you diff the file are the missing events located in once place?
  • Did you start the tests multiple times with the same target file, if the file didn't not change inode, Filebeat would keep an offset or previous run in the data/registry file and will start reading from that and potentially miss events.

Thanks

3

  1. From the 1.000.000 lines, filebeat output files (which is 7 files with 10MB size each), have from 79117 line until 1.000.000 line. That means that the line 0 until 79116 are missing from filebeat.

  2. The events (actually are just lines with a simple txt and a counter) are located only in one file, called main.log. Then, filebeat produces 7 output files, with each file size almost 10MB.

  3. Each time i reproduce the same process, i always delete the data/registry file. And each time, always delete the target file (main.log).

4

@Steve1 This expected behavior, the file output rotates and replace files, by default Filebeat only keeps 7 files and will overwrite older files.

The maximum number of files to save under path . When this number of files is reached, the oldest file is deleted, and the rest of the files are shifted from last to first. The number of files must be between 2 and 1024. The default is 7.

If you look at Configure the File output | Filebeat Reference [8.11] | Elastic

Also note that that by default we rotate at every 10MB.

The maximum size in kilobytes of each file. When this size is reached, the files are rotated. The default value is 10240 KB.

So if you want to test that filebeat persists all your events to disk you will need to play with the above options.

5

Thank you. Honestly, just activating these 2 parameters, my problem got solved.

Really thanks for the help. Just increased the number_of_files: 25 (from default 7), and worked!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.