Read filenames using logstash

Set the document id as a fingerprint of the fields? See this thread for some ideas on how to do that.

If you do not care about the contents of the file (and you say you just need the name) then using a file input plugin and discarding the contents seems rather wasteful. However, periodically running an ls has the same problem.

You want to be able to say whether you have see the file before. An ignore_older on the file input might help, but there will be a window for duplicates when logstash restarts. If you do not care about overwriting old entries then that may not be a problem.

If I were doing this on Windows I would write a PowerShell script that listed all items in a directory tree since the last time it had been run, then pipe that into something that could inject into a logstash input. I cannot think of a similar scripted design on Linux off the top of my head.