How logstash takes new file and sends it to elasticsearch

Elastic Stack Logstash
1

Hi ,

I have a script to create a file that logstash takes as input, and i configured logstash with path for the input file like this ./logs/*/allfiles_decode.json

My concern is ,
for input file creation my script takes 10 mins to complete the write operation,
If logstash takes input file before the script complete the write operation or not ?

2

By default, Logstash would start reading the file as soon as it is created and found by Logstash. If your file contains multiple lines, it will read them line by line anyway. If you append to an existing line, I guess you would need to try that out?

3

Thank you bennyInc,

Ok i do the experiment, i think its wrong to read the file which is in progress (I have not close the file)

4

Logstash is designed to tail files that are constantly appended to, which is the behaviour you are seeing. If you do not want it to start processing the file until it is complete, why not write it in a temporary directory and then move it once it is complete so Logstash can find it at that point?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.