How can filebeat collect logs current date?

Elastic Stack Beats
1

Hi everyone, I am new in ELK Stack, I am encountered problems while collecting logs via Filebeat. Specifically I want to collect logs by path for example: /var/logs/<current_date>/app.log and how can I collect current date and ignore previous dates?

image
image1447×244 27.2 KB

2

Hi, anybody can support me?

3

Does this mean you only want to collect only from the last day, the last week or how?

4

Hi Rios,
thanks for replying, I want to collect logs only current date.

5

You want it to change dynamically? If so, this is not possible.

When you configure a filestream input in filebeat you need to specify the path, you can pass the full path, like /var/log/2024-07-26/app.log or use globs like /var/log/*/*.log, but you cannot configure it to be dynamically change the date.

If on 2024-07-26 you want to get logs only for the path /var/log/2024-07-26 then you need to configure your path without globs in this part, something like /var/log/2024-07-26/*.log.

But to get logs for tomorrow you would need to change the file tomorrow and restart filebeat (all this can be automated outside filebeat with other tools).

You also have the ignore_older option that tells filebeat to ignore files older than some specific time when starting, but this may not work as you want.

This is based on the time filebeat starts (or restarts) and if somefile is added to some older folder filebeat would harvest it.

So, to get logs only for the current date you need to have the current date in the path when configuring your input.

1 Like
6

Leandro told you main point of FB is the feature: ignore_older. Maybe you can use the script processor to drop events if they are not matched by timestemp or date in the directory path.

What you can consider is to involve LS which more powerful for data transformation, where is easy to add a control mechanism to drop all events which are not from today's directory.

There is also the age LS plugin, which was discussed on the topic however it's based on the timestamp field not directory.

1 Like
7

Thank Leandro and Riot so much, Perhaps i need to investigate more.
Thanks