How to use filebeat to collect only the day's logs

Elastic Stack Beats
1

I'm using filebeat to collect logs.
The name of the target log is "api-yyyy-mm-dd.log".

If you set the paths setting of filebeat as follows, all files will be collected.

- type: log
  paths:
    - /var/log/api/api-*.log

How can I configure it to only collect files from the same day?

2

I don't think there is a way to do that but it sounds like the application will only write to one file per day. Is that right? If so once filebeat completes reading the file, it won't read it anymore and will move on. Are you having an issue with the current setup?

3

Thank you for your comment.

The problem often arises during implementation.

As you point out, the target log creates one file per day. There are two months of files with the creation date as the file name.

Once the file is read, it is not read, but on the first day of installing filebeat, it tries to import all the logs in the folder.

I thought it would be nice to be able to collect some scrutiny during input.

4

Gotcha, ur trying to not import the old log files and only import everything from the day u install going forward? Can u not rename the old files to something so it doesn't match the pattern?

5

Yeah. Do you think it would be smart to do so?

If we make app.log for the day, and app-yyyy-mm-dd.log for the rotation, will that accomplish the job of logstash?

- type: log
  paths:
    - /var/log/api/api.log
6

Now we deal with it this way at the beginning of the month.
We think it is not very smart.

- type: log
  paths:
    - /var/log/api/api-2021-04-*.log
7

I guess u could do that, but the registry should track everything so even if u reinstall or restart filebeat, as long as the registry is persistent it should reread it

8

Can't I use the exclude_files option?

It seems to be used to exclude file extensions.

exclude_files: [".gz$"].

However, the following had no effect

exclude_files: ["api-2021-02-*.log", "api-2021-03-*.log"].
9

Is it possible that "*" can be used for paths settings, but not for exclude_files?

It will work if you set it as follows

exclude_files: ["api-2021-(02|03)-(0[1-9]|[1-2][0-9]|3[0-1]).log"]
10

Idk if the exclude files is using globs or regex... to do it's filtering, I'd have to look at the documentation. But that still doesn't seem manageable in the long term. If nothing else works, I think the idea u had about having the file rotation change the name would probably be best long-term since it's automated.

11

I think the idea u had about having the file rotation change the name would probably be best long-term since it's automated.

I have a feeling your idea is correct.
I will try to adjust the file name to see if I can change it.

12

THe only issue with the log name rotation is if Filebeat is still reading the file when it rotates it can cause issues. It was in another thread.

13

That's bad news for me.

Given the nature of filebeat, I guess I shouldn't manipulate files while they are being read.

14

If u use the default naming convention where the file names don't change and u get a new file each day, the registry should prevent previously read files from being reread even on restart and reinstall (provided the registry persists reinstall/update).

15

I see.
I was not aware of the registry.
Can you tell me if this is a filebeat degree setting?

16

See How to understand "registry" file in filebeat - #2 by pierhugues

17

Thanks for adding the explanation of the registry.

I don't think there is any need to rewrite offset in cases where the number of files to be read is newly increased (i.e. logs in the format yyyy-mm-dd.log, etc.), because the same file name will not change the reading position.

In the case where the file name is changed due to log rotation (i.e., the log for the day is today.log, and the next day it is renamed to yyyy-mm-dd.log, etc.), it is not necessary to change the reading position of today.log. In this case, you need to change the reading position, so you need to change the offset and FileStateOS. In this case, you need to change the offset and FileStateOS.

However, I don't know how to set these.
In many operations, it only says to delete the registry if you want to re-read a log that has already been read. The only thing it says is that

I think it is a common case to rename files for log rotation, etc. What are the common means of doing this?

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.