Problem with sincedb

Elastic Stack Logstash
1

Hello,
I'm using the file input with Logstash - just plane text files (pipeline processing) output to elasticsearch and without any filters.

Sometimes I have an issue, when some files have processed but there are no new lines added to the sincedb until I have restarted the logstash. After restart the new lines just added to the sincedb. It hapens not for all files and not on the same condition.
My script uses the sincedb to be aware which files are ready to remove - that's why It is very important to me.

Can someone please explain, how can I solve this problem?

Logstash & Elasticsearch - 5.2

2

I'm not completely clear on what your issue is. Are you saying that new lines are not being recognized by Logstash after a restart, but the sincedb is being updated anyway?

1 Like
3

Sorry,
if a file has processed, Logstash reports about it in .sincedb by addding the new line with inode, offset ...of the file.

But, sometimes, even if some files have already processed (i see the end of the files in a kafka topic - my output way) Logstash isn't reporting about it in .sincedb.
It looks like Logstash being in stuck with updating the .sincedb.
Only after restart Logstach is reporting about all processed files in sincedb.
#processing
file 1 - processed - reported
file 2 - processed - reported
file 3 - processed - reported
file 4 - processed - but not reported
file 5 - processed - but not reported
file 6 - processed - but not reported
file N - processed - but not reported
#restart logstash
reported about 4,5,6...N

4

It is possible that Logstash is having this issue. Logstash is not a good fit for a "start up, read files, shut down" use case. In fact, we now recommend users use filebeat instead for file tailing. But filebeat isn't likely to be a good fit for what it looks like you are doing.

The bottom line is that neither Logstash nor filebeat are meant to batch-read entire files and shut down until more files arrive. They are both designed to run continuously.

I think you will be better suited with a custom script or tool for this use case.

5

Ok, thanks a lot

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.