How to figure out at which line Logstash is in a file?

Elastic Stack Logstash
1

How can I figure out which file is currently processed by Logstash, and at which position Logstash actually is.

I have configured 5 files for input, and usually its working fine. But since hours no one writes to those files - but still I see some Logstash workers using considerable amount of CPU. So I'd like to figure out WHAT Logstash actually does at the moment.

2
3

The scripts in the stackoverflow post wont work for me.

I have some sincedb files with the same inode in it. And with different offset values:
.sincedb_26675a6da459b5bb8f0ec1de6ad0b97f:538586 0 2049 26078748
.sincedb_e4d8738c2308107cbfedd0dfa66b800b:538586 0 2049 100350

Is this OK?

Ty the way: The script from stackoverflow returns this error:
join: /var/lib/logstash/plugins/inputs/file/.sincedb_26675a6da459b5bb8f0ec1de6ad0b97f:4: is not sorted: 543354 0 2049 3033

4

By default the sincedb files are named based on a hash of the filename pattern, so you've probably used different filename patterns that look at the same file. You should probably use the sincedb_path option to explicitly select the path to the sincedb file so you know which entry to use.

5

I have extended that StackOverflow script quite a bit by obtaining the current position that Logstash has read from by using 'lsof'.

My script can be found here: Logstash Progress - Pastebin.com
You'll have to modify the FILES_TO_BE_PARSED and SINCE_DB_FILES variables to point to the correct locations/format that your files are stored.

And example output of my script would look something like:

<% Complete>
/var/log/remotelogs/imsigw-imsigw4.log 97.6% 1592548279 1553569508 524299
/var/log/remotelogs/ussdgw-ussdgw1.log 42.1% 6202389785 2612068352 524300
/var/log/remotelogs/imsigw-imsigw4.log.1 68.8% 1276445987 878379008 524307
/var/log/remotelogs/imsigw-imsigw3.log 93.8% 1592752501 1494754037 524310
/var/log/remotelogs/ussdgw-ussdgw7.log 96.8% 4588987770 4441003272 524701
/var/log/remotelogs/ussdgw-ussdgw1.log.1 40.6% 5152213263 2092012450 524298
/var/log/remotelogs/imsigw-imsigw3.log.1 100.0% 1278359148 1278359148 524304
/var/log/remotelogs/ussdgw-ussdgw7.log.1 80.0% 3830505584 3066234297 524308
Total Summary: 17416380072/25514202317 = 68.3%

I am currently using this script within the sebp/elk docker container, but I had to install 'lsof' first.

I really hope that the issue with regards to the sincedb that has been raised multiple times would get addressed/fixed some time: Sincedb not written when EOF not reached yet

1 Like
6

This is awesome!!

Thanks a million times. This script should be included in every Logstash distribution.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.