Logstash rereading the files even after since_db has that file entry which was already read

Elastic Stack Logstash
1

Hi Team,

I am trying to read the files from a particular path using logstash file input plugin and i am using file_completed_path and since_db path to track the list of files which are processed. It was working fine and logstash is continuously updating both the file_completed_path, since_db with the files and reading only new files. But suddenly from today it started reading the old files and creating a duplicate entries for the same files in file_completed_path, since_db path.

There are no modifications to file_completed_path, since_db paths and using them from starting. Could you please suggest what may be the route cause for this issue and why logstash is reading the files again even they are already read and updated in the since_db path.

Thanks in advance.

2

Can you show us an example of a duplicated entry?

3

Hi @Badger,

Please find the below sample entries in the file_completed_log_path folder for my pipeline. The same file has read twice. One read happened on Nov 12 and another one Today(Nov 18).

/var/log/Elastic/sas_data/sas_elastic_orders_dump_20241112.csv
/var/log/Elastic/sas_data/sas_elastic_orders_dump_20241112.csv

4

I think you need to share the duplicate entries in the sincedb file, showing the inode as well.

Is /var/log/Elastic/sas_data a NFS share? What is creating those files? Also, share your file input as well.

5

Hi @leandrojmp,

Please find the below entries from since_db file.

6450506898 0 41 1432982 1731840644.927283 /data/apac/nfs/elk/sas_elastic_orders_dump_20241112.csv
6450506898 0 40 1432982 1731922241.158001 /data/apac/nfs/elk/sas_elastic_orders_dump_20241112.csv

Yes, we are getting these files from NFS share which is mounted on our logstash server.

File input plugin:

input {
  file {
     id => "sas-orders-input"
     path => "/data/apac/nfs/elk/sas_elastic_orders_dump_*.csv"
     mode => "read"
     file_completed_action => "log"
     file_completed_log_path => "/var/log/Elastic/sas_data"
     start_position => "beginning"
     sincedb_path => "/var/log/Elastic/db_logstash/sas_data"
        }
}

Thank you in Advance.

6

The device number has changed from 40 to 41, which means logstash thinks it is a different disk. That can happen with NFS mounts when they are re-mounted (e.g. after a reboot). There's nothing logstash can do about it.

You may be able to fix it for the client/server (e.g. here) but that will be OS specific.

1 Like
7

Hi @Badger ,

Thank you for the update. Now the NFS team is planning to migrate the FS to Azure NFS after they did the changes again will our logstash reprocess all the files. Is there any way to handle this situation without rerunning the old files.

We are not creating any lvcreate to create volumes we are just mounting it in our server using fstab.

XXXX:/data/ /nas_data nfs defaults 0 0

Our OS Details:
RHEL 7.9(mapio)

8

I don't think you can affect the device numbers using fstab.