I'm successfully using logstash to parse json formated events and send them to elasticsearch. Each event is created in a seprate file. One event per file with .json extension.
Logstash is correctly picking up a file when i create it using "vi mydoc.json", paste in the content and save. However it does not pick up if I cp or mv a file.
The objective is to automatically copy files to a directory and then parse them by logstash.
Each file has a different name and size. I tried looking at logstash code to figure out what attribute it uses but couldnt find relevant code. I also tried deleting the .sincedb files but didn't help either.
The input config is as follows:
path => "/opt/rp/*.json"
type => "tp"
start_position => "beginning"
stat_interval => 1
How can I have logstash pick up copied files? What file stat attribute does it use to check if a file is new?