Sincedb minor device number changes in Linux VM | alternative sincedb configurations options?

Elastic Stack Logstash
1

Hi,

I have a Logstash -> Elasticsearch set p. Everything is working as expected, but I have a problem with the sincedb or better with my Linux VM underneath.

If I reboot the system, the device number of the partition is changing.
So Logstash thinks it has to import the log file again.

For better understanding here my setup
My pipeline.conf with the sincedb setup

 input {
  file {
    path => [
              "/usr/share/logstash/logs/localhost_access_log*.gz"
            ]
    start_position => "beginning"
    mode => "read"
    sincedb_path => "/usr/share/logstash/sincedb/my_sincedb"
  }
}
...

sincedb content after first import

151074174 0 64769 41343 1710406427.02369 /usr/share/logstash/logs/localhost_access_log.2023-09-30.txt.gz

And after the reboot a cleary sign that the device number has changed

151074174 0 64769 41343 1710406427.02369 /usr/share/logstash/logs/localhost_access_log.2023-09-30.txt.gz 
151074174 0 64772 41343 1710481076.099891 /usr/share/logstash/logs/localhost_access_log.2023-09-30.txt.gz

I can't do anything about the vm configuration atm because it is not in my hand but if I call stat -c "%D" /usr/share/logstash/logs/localhost_access_log.2023-09-30.txt.gz

I can clearly see the same changes to support my theory :wink:

Are there any other ways to set up sincedb to rely on something other than the minor device number? Are there any other options I have?

Many thanks in advance for any hints.

2

Is this path a network share or it is a path inside the vm disk?

3

To be honest, I don't know exactly what type it is atm.
I will try to find out. My first guess is vm disk.
What would be the difference in context of the device number?

If I do a stat on some files before and after reboot this is the stat output

stat  /usr/share/logstash/logs/localhost_access_log.2024-01-24.txt.gz
...
  Size: 14469           Blocks: 32         IO Block: 4096   regular file
Device: fd04h/64772d    Inode: 151673571   Links: 1
Access: (0775/-rwxrwxr-x)  Uid: ( 1000/logstash)   Gid: ( 1000/     logstash)
Access: 2024-03-15 00:46:03.241302011 +0100
Modify: 2024-01-24 07:44:55.607909195 +0100
Change: 2024-01-26 06:35:02.028559558 +0100
 Birth: 2024-01-26 06:35:02.014559520 +0100

stat  /usr/share/logstash/logs/localhost_access_log.2024-01-24.txt.gz
...
  Size: 14469           Blocks: 32         IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 151673571   Links: 1
Access: (0775/-rwxrwxr-x)  Uid: ( 1000/logstash)   Gid: ( 1000/     logstash)
Access: 2024-03-15 00:46:03.241302011 +0100
Modify: 2024-01-24 07:44:55.607909195 +0100
Change: 2024-01-26 06:35:02.028559558 +0100
 Birth: 2024-01-26 06:35:02.014559520 +0100
4

No, filewatch assumes that the same device will have the same device number, and uses that to track the file, not the name.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.