Does Logstash process single file or multiple files simultaneously?

How does the logstash process multiple log files ??? What I mean is, does the logstash processes one file at a time (and then move on to another file) or multiple files simultaneously ???

Logstash was designed to monitor and process files as they are actively being written to, so it will process multiple files simultaneously.

So, why does the .sincedb file is updated only when a file is completely processed ?

I am using logstash to process the logs (not to monitor it).

I had put 2 log files of size 100 Mb each in the folder being monitored by logstash. Information about first file was entered in .sincedb file only when logstash finished processing it. Same thing happened with second file.

if you really want to process logs from multiple files on the Go, meaning if file 2 updates often then file1, Better use filebeat. it monitors multiple files, and send responses to logstash .

So, why does the .sincedb file is updated only when a file is completely processed ?

I don't know, but it's a different question from "will Logstash process files in parallel".

I had put 2 log files of size 100 Mb each in the folder being monitored by logstash. Information about first file was entered in .sincedb file only when logstash finished processing it. Same thing happened with second file.

I think Logstash processes files in a given file input serially but if you have multiple file inputs the files will be processed in parallel.

Will I be able to get the Logstash reading progress of each individual file while the file is being processed ? (I have not used Filebeat before)

I will take care of this next time. :slight_smile:

Will I be able to get the Logstash reading progress of each individual file while the file is being processed ?

Filebeat keeps a registry file that's similar in spirit to Logstash's but in a different format.

Yeah Mayank. File beat will monitor all logs included in the yml. But for first run it also spans the logs sequentially. After that , any update in listed files will be pushed to logstash. in logstash u need to add a filter for each file incase the GROKs are different for each file.

Yeah I saw the registry , its similar to logstash. but for first run , I guess its always sequential spanning files is a apt solution. beat monitors all files and push any new lines from last read.