File input not picking up copied or moved files

Elastic Stack Logstash
1

Hi,

I'm successfully using logstash to parse json formated events and send them to elasticsearch. Each event is created in a seprate file. One event per file with .json extension.

Logstash is correctly picking up a file when i create it using "vi mydoc.json", paste in the content and save. However it does not pick up if I cp or mv a file.

The objective is to automatically copy files to a directory and then parse them by logstash.
Each file has a different name and size. I tried looking at logstash code to figure out what attribute it uses but couldnt find relevant code. I also tried deleting the .sincedb files but didn't help either.

The input config is as follows:

input {
file {
path => "/opt/rp/*.json"
type => "tp"
start_position => "beginning"
stat_interval => 1

  }

}

How can I have logstash pick up copied files? What file stat attribute does it use to check if a file is new?

Thanks

2

Logstash's file input (actually the Ruby filewatch library) tracks files via inode numbers. Copying or moving a file into a directory should make Logstash treat it as a new file unless there's inode reuse going on. You'll get additional debugging out if you start Logstash with --verbose.

3

Even I am facing the same issue where Log Stash is able to pickup the file which I created there but unable to pick up the files which is there by either copying or Moving or from Downloading.

Suggest some points.

4

Perhaps the files are old than 24 hours and you haven't adjusted the file input's ignore_older option. Otherwise the previously given advice of using the --verbose option is still relevant.