Filebeat: how to define a date pattern fo prospector's path?

Elastic Stack Beats
1

Hi,

I have logs which are named like mylog-20170531.log.
The logs are pre-rotated at midnight UTC.

If I define path/mylog-*.log It would read all logfiles, regardless of the date, if they have been created / modified within a specific time.

Is there a way to explicitly define something like /path/mylog-YYYYMMDD.log ?

Thanks, Andreas

2

Hi @asp,

As far as I know there is no way to filter that much, paths field supports glob like matching: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#prospector-paths

Something you could do is creating a symlink to todays log file and read always from that one, what do you think?

3

maybe some more details here are interesting for you. Maybe I do not really have a problem at all.

background:
Our application has a problem, if it needs to append to a log > 4GB when starting. If the application is already running and then it gets over the 4GB limit, there is no problem.

So our workaround is to rename the old log, then the application is creating a new logfile, which will be sipped to logstash.

I need to prevent filebeat to ship the renamed file to logstash, because the data is aready in elasticsearch.
Or do I not need to worry about it? In the past we always managaed to rename to *.log_, so it did not match the pattern at all, but I would like to build a waterproof setup.

symlinking may be possible, but I think we will stay on the order to rename by adding a suffix to the logfile name which is not matching the pattern.

4

If you control file names best thing to do is rotating into paths that do not match the regexp, filebeat should ignore those

5

good idea, unfortunately this would disrupt our backup strategy of logs on filesystem level :wink:

But I think with adding a suffix which breaks the regex, it should be fine too. Thanks

6

This topic was automatically closed after 21 days. New replies are no longer allowed.