.,..
I think I can use logstash with file input plugin and tweak multiline codec to read whole file as one entry and then use "path" in grok fliter and read all required data.
I can also use a shell script to first read all files in directory and put them in a different file. Then use logstash to send that file data. Or directly write a shell script to send data directly to ES.
My folder is regularly getting updated with new files and I want those filenames to go in to ES.
Can anyone else suggest me a better way to do this with logstash or any other suggestions?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 20, 2018, 6:26pm UTC](https://discuss.elastic.co/t/read-filenames-using-logstash/124758/2 "2018-03-20T18:26:28Z")
Use an exec input to periodically run `ls /path/to/directory`.
---
### Author:  [@Mahi](https://discuss.elastic.co/u/Mahi)
#### Post date: [March 20, 2018, 11:07pm UTC](https://discuss.elastic.co/t/read-filenames-using-logstash/124758/3 "2018-03-20T23:07:18Z")
Thanks Magnus..but problem with this is that ls will return all the filenames every time I execute it..but I want only new filenames..
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 21, 2018, 12:26am UTC](https://discuss.elastic.co/t/read-filenames-using-logstash/124758/4 "2018-03-21T00:26:08Z")
Set the document id as a fingerprint of the fields? See [this thread](https://discuss.elastic.co/t/logstash-fingerprint-plugin-how-to-exclude-timestamp/112498/9) for some ideas on how to do that.
If you do not care about the contents of the file (and you say you just need the name) then using a file input plugin and discarding the contents seems rather wasteful. However, periodically running an ls has the same problem.
You want to be able to say whether you have see the file before. An ignore\_older on the file input might help, but there will be a window for duplicates when logstash restarts. If you do not care about overwriting old entries then that may not be a problem.
If I were doing this on Windows I would write a PowerShell script that listed all items in a directory tree since the last time it had been run, then pipe that into something that could inject into a logstash input. I cannot think of a similar scripted design on Linux off the top of my head.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 18, 2018, 12:26am UTC](https://discuss.elastic.co/t/read-filenames-using-logstash/124758/5 "2018-04-18T00:26:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.