How to parse today's file

vikas_gopal · July 21, 2015, 9:37am

Hi Experts,

I want parse today's log file . I tried the following (kind of stupid thing I have done as it works only for output )but no luck .Please suggest how I can achieve this .

input {
file {
path => "E:\CEF\SIEM-%{+YYYY-MM-dd}.txt"
start_position => "beginning"
}
}

magnusbaeck · July 21, 2015, 9:49am

See the recent thread below. I don't think %{+YYYY-MM-dd} works in this context since it uses the current event's timestamp, but inputs don't have a current event context.

vikas_gopal · July 21, 2015, 9:59am

so does that means if I have 30 files in my folder (I mean one for each day), Logstash will read all those again because we will be using * .I understand sincedb will store a pointer but that would be very hectic for LS ?

magnusbaeck · July 21, 2015, 10:06am

so does that means if I have 30 files in my folder (I mean one for each day), Logstash will read all those again because we will be using *.

With start_position => "beginning" the files will be read again if they are new to Logstash.

I understand sincedb will store a pointer but that would be very hectic for LS ?

Logstash will stat all files matched by the configured pattern every stat_interval seconds. For 30 files that shouldn't be a problem, and as long as the files aren't updated the sincedb file won't be affected.