Determine when Logstash is done parsing

Elastic Stack Logstash
1

I want to know when Logstash is done parsing. I know that information can be found in this link:

But am unsure of where I put the code in the logstash.conf file? Also, in the comment states this:

Unfortunately when you're using the file input file with 'start_position => "beginning"' with Logstash 5, it does not write anything to the sincedb file until its done - or at least this is the behaviour that I'm getting.

I am using the start_position => "beginning" with the latest Logstash. Here is a sample of what my input looks like.

input{
	file{
		path => "/opt/ELK/data/**/admin_ui*.csv"
		start_position => beginning
		ignore_older => 0
		sincedb_path => "/dev/null"
		type => "csv"
	}
}

There will be a definite end to the files that need to be parsed because the files will be placed in the /data directory before Logstash is started. Any tips?

Thanks!

2

But am unsure of where I put the code in the logstash.conf file?

What code?

There will be a definite end to the files that need to be parsed because the files will be placed in the /data directory before Logstash is started. Any tips?

Since the idea being discussed in that StackOverflow post relies on reading the sincedb file you need to stop disabling sincedb. The comment about sincedb not being written to until the file is done doesn't matter in your case since you only seem to care about whether the file has been processed or not (rather than getting a "percent completed" figure).

3

The code that I am talking about is in your comment:

$ join /var/lib/logstash/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc <(ls -li /var/log/syslog) | awk '{ printf "%-30s%.1f%\n", $13, 100 * $4 / $9 }'
/var/log/syslog               100.0%

And I am not sure what you mean by disabling the sincedb. It currently writes to a file called null and uses it as the sincedb file. Is this an issue?

4

The code that I am talking about is in your comment:

That's an example command that you can run in the shell of most Unix-like operating systems.

And I am not sure what you mean by disabling the sincedb. It currently writes to a file called null and uses it as the sincedb file. Is this an issue?

Yes, technically it's writing a file named "null" but nobody can ever read anything from that file. Your current setting of sincedb_path effectively disables sincedb.

5

I am running the ELK stack on Windows. Is there any Windows commands that would work the same as the one you used above?

6

I'm sure there's something equivalent in PowerShell but I wouldn't know what it looks like. But again, that was just an example. The main point is the you can read the sincedb file and compare the current offset number with the file size to figure out whether Logstash is done.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.