Logstash gzip_lines , tail latest written file to text file

[shashwat]

Hi,

I am using logstash 5.5. I am using gzip_lines plugin to parse the archived files. As , this plugin expects a text file having archived file with absolute path line by line. If this text file is static , i.e. if I write path for all the archived files in it and run logstash it works fine.

BUT , when I write this text file at the run time every minute with the new path of archived file, logstash does not process the latest file being written in the .txt file. I tried with start_position="end" but that did not help. If I use start_position="beginning", it re-process all the files written in the txt file and that I don't want.

I just want to process latest file being written in the txt file. Is there any way to achieve this?

Thanks,

[magnusbaeck (Magnus Bäck)]

You mean you're updating the gzipped log file while Logstash is running? Why? I wouldn't expect that to work.

[shashwat]

Hi Magnus,

I have these .csv.gz files coming every minute. I need to analyze them using Kibana. As per my understanding and the information I found in different forums, if we want to use gzip_lines plugin, we need to have a .txt file with the path of .gz files in it . That's why , I was trying to update the .txt file every minute with the path of the received .gz file.
However; another way of solving my problem can be by extracting the received .gz files in some location and then parse it using logstash/Kibana.

Thanks,