How to add a grok expression or regex in the file name of a log for parsing

Elastic Stack Logstash
1

Hi

I am trying to parse a logfile which is name as testlogfile.YEAR-MM-DD.log

I used the input plugin of file with the format as testlogfile.%{+YYYY-MM-dd}.log

This is not working and the only way to get this file read is by testlogfile.*.log

Since this is a rolling log created for each day, I want to get the file name based on the regular expression. Is there any way around getting this working ? Also will this work with logstashforwarder ?

2

I used the input plugin of file with the format as testlogfile.%{+YYYY-MM-dd}.log

This is not working and the only way to get this file read is by testlogfile.*.log

Yes. I believe the %{+date format} syntax formats the event's timestamp so it won't work for inputs.

Since this is a rolling log created for each day, I want to get the file name based on the regular expression. Is there any way around getting this working ?

So you think testlogfile.*.log is too broad and you want to restrict Logstash to only read files named testlogfile.YYYY-MM-DD.log? You can't use a regular expression, but you can use a ? wildcard so you could e.g. use testlogfile.20??-??-??.log as the filename pattern.

3

Hi Magnus

I can go with your suggestion but was wondering as to how the offset works so that it is only picking the new log entry in file as well as newly created files so that old logs lines are not available for parsing
Any link where I can get this info ?

Rajeev
Sent from my iPhone

4

Have you checked out the file input documentation? For previously unseen files Logstash will, with the default setting of start_position, tail the file (i.e. only read new entries). For known files where an offset has been saved it'll obviously start at the saved offset.

5

Thanks for the clarification
Also I think the logstash forwarder doesn't ship the logs which are more than 24 hours old
Just need to fix the ACK error which is coming because of the spool size and timeout