Order of file generation and file reading


#1

I used filebeat logstash and elasticsearch together to port data from files to elasticsearch, I have a process to generate files in a folder, and filebeat reads them and sends data to logstash then to elasticsearch. The flow goes like this:
files (step 1) -> filebeat(step 2) -> logstash (step 3)- elasticsearch

It seems to me the order is very important, I need to start the application to generate files(step 1) first, and then start filebeat (step 2)

If I do it in an opposite order, start filebeat process first, and then start generating files , filebeat cannot read files. I used -e -d "*" to track the response when initiating the filebeat, the process responds like "filebeat is harvisting on the file". But it will never start reading the file.

So shall I assume that I have to start generating the files before starting filebeat to read the files?


(Mike Simos) #2

Hi,

Using a path like /path/to/files/* the prospector will scan the directory for new files every 10 seconds. So you shouldn't need to generate the files before starting Filebeat. You can also decrease the scan_frequency if you want Filebeat to check for new files quicker:

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#_scan_frequency


(system) #3