Logstash processing the same file again and again from a remote shared path

(Anirban Sen) #1


I have a logfile in a shared path (//XXXX/storage/files/test.log) with 100 lines.

I am processing the it with logstash with the below input cong:
input {
file {
path => "//XXXX/storage/files/test.log"
start_position => "beginning"

But, logstash is processing the same 100 lines of the file again and again.
I want it to stop after it processes the file once.

Please suggest.

(Colton) #2

You might want to take a look at the since_db setting, which you can read about here

(Anirban Sen) #3

Hi @CDR , tried with different options, but no success. Could you please suggest which since_db option need to be used.

(Colton) #4

sincedb_path is what I would suggest. Point it to a location and it will create a text file that keeps track of the location of the location in the file it has read.

(Anirban Sen) #5

Hi @CDR, I added this:

input {
file {
path => "//XXXX/storage/files/test.log"
start_position => “beginning”
sincedb_path => "D:/sincedb/test.txt"

But, still same issue. Logstash keeps on reading and processing the same 100 lines from the file though it is not modified.

(Christian Dahlqvist) #6

Reading files from a shared path is known to have the potential to result in issues, as outlined in the documentation.

(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.