How can i configure logstash to read every new csv files automatically?

Elastic Stack Logstash
1

How can i use logstash to read csv files in real time ? How can i configure logstash to read evey new csv files automatically ?
I have an empty folder,I want each time by adding a file in this folder, logstash automatically read the data in real time and I also want to know if I have a lot of csv files how logstash can know the processed files or the files which have problems for example
Any help please??

2

setup your input like this and it will read it all file with that name gets dumped in to that folder

input {
file {
path => "/data01/csv_files/*.log"
sincedb_path => "/dev/null"
start_position => "beginning"
}

1 Like
3

Thanks for the reply, i just want to understand something please, for example i have:
file01.csv
file02.csv
......
file10.csv
if there are problems at file02.csv level how logstash will inform me that there are problems or if i stop it and i start it another time does it know which file is stopped .
another thing is can i view the daily data reception status with logstash or kibana?

4

it depends on what kind of problem.
if you file as some bad data and if logstash does not read it then it won't inform you.
there is no mechanization to alert you.

Once it reads the file it won't re-read it. it knows it has read it.
if you delete the file file10.csv and next day if you copy file with same name it will re-read it as it goes by files inode#. as long as it is different inode it will read it, considering it has same file name file*.csv

1 Like
5

Thank you Sachin for these informations :blush: :wink:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.