I used filebeat logstash and elasticsearch together to port data from files to elasticsearch, I have a process to generate files in a folder, and filebeat reads them and sends data to logstash then to elasticsearch. The flow goes like this:
files (step 1) -> filebeat(step 2) -> logstash (step 3)- elasticsearch
It seems to me the order is very important, I need to start the application to generate files(step 1) first, and then start filebeat (step 2)
If I do it in an opposite order, start filebeat process first, and then start generating files , filebeat cannot read files. I used -e -d "*" to track the response when initiating the filebeat, the process responds like "filebeat is harvisting on the file". But it will never start reading the file.
So shall I assume that I have to start generating the files before starting filebeat to read the files?