File watcher

Elastic Stack Logstash
1

Hello all,

I have a problem regarding file input to get data from several csv files.

The goal is pretty simple but i'm struggling.
I need to retrieve data from a csv file, easy, I works nicely. The data are stored in ES.

But, I need more than that. In fact, let's say I have my csv file under:
/data/patch-management/my_file.csv

Every now and then, a new version of this file is send to my folder /data/patch-management/ under the same name as the previsous csv file.

So it will be erase with new version of this file.
That being said, I now have my new csv file under:
/data/patch-management/my_file.csv

But contain different information from the first one.

What I need: To erase the file once its read by Logstash, and then read the new file. But I don't know when it will be "update", so my Logstash need to "watch" the folder, and if it sees that the new file is not read (thanks to sincedb_path, I guess), reads it, and store the content in my ES index (the same as the first file).

Example: My first file as 4 lines, then, my ES index has 4 events.
My second file as 6 lines, then my ES index should have 10 events.

I put a "file_completed_action" => "delete", which, once read, will delete the file "my_file.csv". It is not the case. Logstash have access to this file so it is not a permission problem.

Here's my config:

input {	 
  file {
    path => "/data/patch-management/*.csv"
	sincedb_path => "/etc/logstash/conf.d/since_db_path/patch-management_csv.sdb"
	mode => "read"
	file_completed_action => "delete"
	type => "patch_management"
  }
}

The filter and output works great, no problem on this part.

I hope someone can explain how to achieve this.

Thanks in advance.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.