Logstash 8.3.3 removes entries from sincedb

Elastic Stack Logstash
1

Hello,

I have a pipeline using Logstash 8.3.3
The pipeline is consuming csv file from an input folder using file input
The csv files are produced by an other system on the flight

This my input

input {
  file {
    path => ["/u01/*.csv"]
    sincedb_path => "/opt/tmp/sincedb_file.txt"
    file_completed_action => "log_and_delete"
    file_completed_log_path => "/opt/tmp/processed_files.txt"
    mode => read
    ecs_compatibility => "disabled"
    file_sort_by => "last_modified"
    file_sort_direction => "asc"
    max_open_files => 10
    start_position => "beginning"
    sincedb_clean_after => "356 d"
    exit_after_read => false
  }
}

Logstash is processing files and sending documents into Elasticsearch.
The producer send 1984 csv file into /u01
After a while, i can see 1984 entries in /opt/tmp/processed_files.txt
But only 98 entries in /opt/tmp/sincedb_file.txt

Did any one have an explantion to this, i'm supposed to have 1984 entries in /opt/tmp/sincedb_file.txt when all files are processed and deleted

Thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.