How logstash forwarder works?


(sampasei) #1

Hi all,
i have a question about logstash-forwarder.
I'm using a log program able to write txt files that are zipped and created again every hour: consider this case

8:59 my program is writing a file log.txt
9.00 log.txt is zipped and archived, a new log.txt is created

Logstash-forwarder hasn't a buffer: its buffer is the log file. Consider this case:

8:50 logstash on the server side is down, logstash-forwarder tries to reconnect without any result
9:00 logstash on the server side is still down, a new log file is created
9:10 logstash is up, logstash-forwarder sends records to the server

I thought that the first record sent to the server was the first record of the file created at 9:00, but i see that previous records are sent to the logstash server....why?
Thanks


(Magnus B├Ąck) #2

I haven't studied the LSF source code in detail, but I'd assume this is because LSF kept the original file open all the time. Recall that in Unix-like systems it's normally fine to delete a file that a process has open without affecting the operation of that process. Once that process closes the file it'll be deleted from disk.


(system) #3