# How to figure out at which line Logstash is in a file?

**URL:** https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759
**Category:** Logstash
**Created:** [January 23, 2018, 9:57pm UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759 "2018-01-23T21:57:13Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Precision](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@Precision](https://discuss.elastic.co/u/Precision)
#### Post date: [January 23, 2018, 9:57pm UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/1 "2018-01-23T21:57:14Z")

</div>

How can I figure out which file is currently processed by Logstash, and at which position Logstash actually is.

I have configured 5 files for input, and usually its working fine. But since hours no one writes to those files - but still I see some Logstash workers using considerable amount of CPU. So I'd like to figure out WHAT Logstash actually does at the moment.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 24, 2018, 5:23am UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/2 "2018-01-24T05:23:01Z")

</div>

> <https://stackoverflow.com/questions/33143437/logstash-parsing-progress-bar>

---

<div class="post-metadata">

### Author: ![Precision](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@Precision](https://discuss.elastic.co/u/Precision)
#### Post date: [January 24, 2018, 6:04am UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/3 "2018-01-24T06:04:01Z")

</div>

The scripts in the stackoverflow post wont work for me.

I have some sincedb files with the same inode in it. And with different offset values:  
.sincedb\_26675a6da459b5bb8f0ec1de6ad0b97f: **538586** 0 2049 26078748  
.sincedb\_e4d8738c2308107cbfedd0dfa66b800b: **538586** 0 2049 100350

Is this OK?

Ty the way: The script from stackoverflow returns this error:  
join: /var/lib/logstash/plugins/inputs/file/.sincedb\_26675a6da459b5bb8f0ec1de6ad0b97f:4: is not sorted: 543354 0 2049 3033

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 25, 2018, 11:02am UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/4 "2018-01-25T11:02:45Z")

</div>

By default the sincedb files are named based on a hash of the filename pattern, so you've probably used different filename patterns that look at the same file. You should probably use the `sincedb_path` option to explicitly select the path to the sincedb file so you know which entry to use.

---

<div class="post-metadata">

### Author: ![padakwaak](https://avatars.discourse-cdn.com/v4/letter/p/eada6e/32.png) [@padakwaak](https://discuss.elastic.co/u/padakwaak)
#### Post date: [January 25, 2018, 6:15pm UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/5 "2018-01-25T18:15:53Z")

</div>

I have extended that StackOverflow script quite a bit by obtaining the current position that Logstash has read from by using 'lsof'.

My script can be found here: [Logstash Progress - Pastebin.com](https://pastebin.com/qsjYKd5W)  
You'll have to modify the FILES\_TO\_BE\_PARSED and SINCE\_DB\_FILES variables to point to the correct locations/format that your files are stored.

And example output of my script would look something like:

> \<% Complete\>   
> /var/log/remotelogs/imsigw-imsigw4.log 97.6% 1592548279 1553569508 524299  
> /var/log/remotelogs/ussdgw-ussdgw1.log 42.1% 6202389785 2612068352 524300  
> /var/log/remotelogs/imsigw-imsigw4.log.1 68.8% 1276445987 878379008 524307  
> /var/log/remotelogs/imsigw-imsigw3.log 93.8% 1592752501 1494754037 524310  
> /var/log/remotelogs/ussdgw-ussdgw7.log 96.8% 4588987770 4441003272 524701  
> /var/log/remotelogs/ussdgw-ussdgw1.log.1 40.6% 5152213263 2092012450 524298  
> /var/log/remotelogs/imsigw-imsigw3.log.1 100.0% 1278359148 1278359148 524304  
> /var/log/remotelogs/ussdgw-ussdgw7.log.1 80.0% 3830505584 3066234297 524308  
> Total Summary: 17416380072/25514202317 = 68.3%

I am currently using this script within the sebp/elk docker container, but I had to install 'lsof' first.

I really hope that the issue with regards to the sincedb that has been raised multiple times would get addressed/fixed some time: [Sincedb not written when EOF not reached yet](https://discuss.elastic.co/t/sincedb-not-written-when-eof-not-reached-yet/116059)

---

<div class="post-metadata">

### Author: ![Precision](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@Precision](https://discuss.elastic.co/u/Precision)
#### Post date: [January 26, 2018, 11:52am UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/6 "2018-01-26T11:52:13Z")

</div>

**This is awesome!!**

Thanks a million times. This script should be included in every Logstash distribution.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 23, 2018, 11:52am UTC](https://discuss.elastic.co/t/how-to-figure-out-at-which-line-logstash-is-in-a-file/116759/7 "2018-02-23T11:52:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
