The subject says it all.
If you aim to store filenames in elasticsearch, maybe you could use this crawler :
And yes, this is not a logstash answer
What's the exact definition of "watch"? Do you want an event if and only if the set of files in the directory changes?
I'd like exactly the behavior of the file plugin except I only want the name of the file.
One nice thing about the file input plugin is that it "remembers" in since_db which files it has already processed.
So for example, I could watch directory /a/b/c/d for files matching the pattern "core.*".
I'm considering moving to polling [osquery] given the rich set of data it can provide to me.
Did you give a try to https://github.com/dadoonet/fscrawler ?
Fabien,
Thank you for taking time to read my question and to make suggestions. I have not had a chance until now to look at fscrawler. It seems that fscrawler will stuff data into ES (+), but I'd rather do some LS style filtering before sending all file updates to ES (-) . If I want to inspect the contents of a file, then LS already does what I want. If I only want to monitor directories for files being added, then (barring an LS plugin), fscrawler using Java seems to be quite a heavy hammer (-). Why wouldn't I just use inotify? My deployment is restricted to Linux and other parts of the toolchain required for my project already depend on inotify being installed and working. With some simple python, I can post events I care about to from a python inotify to an http LS input plugin.
Ok.
Given your context, I agree with you, inotify seems to be a good solution.
And if you can then post Json document from inotify to logstash, this is perfect.