How to figure out at which line Logstash is in a file?

I have extended that StackOverflow script quite a bit by obtaining the current position that Logstash has read from by using 'lsof'.

My script can be found here: Logstash Progress - Pastebin.com
You'll have to modify the FILES_TO_BE_PARSED and SINCE_DB_FILES variables to point to the correct locations/format that your files are stored.

And example output of my script would look something like:

<% Complete>
/var/log/remotelogs/imsigw-imsigw4.log 97.6% 1592548279 1553569508 524299
/var/log/remotelogs/ussdgw-ussdgw1.log 42.1% 6202389785 2612068352 524300
/var/log/remotelogs/imsigw-imsigw4.log.1 68.8% 1276445987 878379008 524307
/var/log/remotelogs/imsigw-imsigw3.log 93.8% 1592752501 1494754037 524310
/var/log/remotelogs/ussdgw-ussdgw7.log 96.8% 4588987770 4441003272 524701
/var/log/remotelogs/ussdgw-ussdgw1.log.1 40.6% 5152213263 2092012450 524298
/var/log/remotelogs/imsigw-imsigw3.log.1 100.0% 1278359148 1278359148 524304
/var/log/remotelogs/ussdgw-ussdgw7.log.1 80.0% 3830505584 3066234297 524308
Total Summary: 17416380072/25514202317 = 68.3%

I am currently using this script within the sebp/elk docker container, but I had to install 'lsof' first.

I really hope that the issue with regards to the sincedb that has been raised multiple times would get addressed/fixed some time: Sincedb not written when EOF not reached yet

1 Like