Date pattern in file input path

(shashwat) #1


I am in need to read files from different folders. These folders are maintained daily in the format of dates. e.g. 20170912, 20170911 etc.

I am trying the file input path as "var/opt/data/%{+YYYY}%{+MM}%{+dd}/*.txt" but it is not working.
How can I read files in such a case?


(Magnus Bäck) #2

There's no automatic way of selecting "file of the day". Can't you just use a wildcard and have Logstash read all files as soon as they become available?


Depending on your platform you might be able to set an environment variable to the name and then reference that in the config. For example, I have a nightly job that processes the prior day index. In the script that runs I set

 INDEXDATE=`date --date="-1 day" '+%Y.%m.%d'`

and in the logstash configuration I have an elasticsearch input that reads

index => "logstash-${INDEXDATE}"

(shashwat) #4

Actually , there are other files/folders also in that directory besides these date formatted folders. I do not want to process all that data. I just need to target these date specific folders. For wildcard , either I give something like /var/opt/data/2017*/*.txt . Can you please suggest any better way?

(Magnus Bäck) #5

If you can find a wildcard that matches the files you want to capture and nothing else I'd say go for it. Otherwise you'll have to periodically generate a configuration file with specific paths or use @Badger's solution with an environment variable.

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.