How to make my filebeat read from the beginning in log file

Elastic Stack Beats
1

How to make my filebeat read from the beginning in log file everytime.

i am using filebeat client node is sending CSV formatted log file to the logstash node and
i have CSV filter in logstash which is creating fields and feeding data to elastic search.
where and how can i mention that it should read the log file from the beginning.

i am looking for the same option like "start_position => beginning" for my scenario.
can someone please help.

2

Hi @rajkumar.m,

Filebeat reads files from the beginning by default, you don't need to set any option for that. tail_files can be used for the opposite behaviour, to read files from the end.

Could you give more details about your problem and about how these CSV files are written?

3

Thanks for reply,

my app will be creating new logs in csv format instead of append to exist.
filebeat sending those log files to logstash in frequency of 15 min.
the issue is filebeat is ignoring the log and not sending as expected.
i thought it is not reading the data as these it is overwriting always.

as a workaround i did my app append data to existing logs then it is working.
but i can't keep the files growing.
please advise.

4

Do you mean that the file is overwritten each time new logs are created? In that case it can be difficult to know when the file has to be read from the beginning.

To avoid this problem and to avoid having log files growing infinitely, the usual approach is to rotate them, that means to move the existing file to a new path from time to time and create a new file in the path you are collecting. There are multiple strategies to implement this, you can use some software like logrotate, or you can implement it in your own application.

Other option that could fit in your case is to write each time to a timestamped file, so every time you write a new CSV it is in a new and different file. Then you can use something like tmpreaper to remove old files. You could add a path with a wildcard in filebeat so it collects from all possible files, take a look to close_inactive and close_eof settings if you try something like this.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.