Reading from a deleted file?

[gcsaba2 (Csaba Galyó)]

Hello,

I'm curious about one scenario on linux. As you know, on linux if I have a file with two processes having a handle to it, then the file won't get deleted permanently until both handles are unlinked. Meaning: if I have process A writing to the file every second and after 10 second it deletes it, and process B is reading from the file every 5 seconds (without ever closing the file handle), then process B will have access to the file even after it has been deleted from the filesystem by process A, and the file won't get deleted until B closes its handle as well.

Now I'm interested in how this applies to Logstash? Let's say I have a process A writing to a text file 100 lines/sec. I have logstash configuration with: input { file { } } and it's going to read from this file let's say every 15 seconds. Is the fike handle going to remain open during the interm period?

If process A deletes the file, and Logstash "activates" 15 seconds later, then will it still have access to the old file? Or will the last 15 seconds disappear?

A real life use case is that we are archiving & deleting log files when some conditions are met, but we don't want Logstash to miss any data due to this.

Thanks,
Csaba

[magnusbaeck (Magnus Bäck)]

Is the fike handle going to remain open during the interm period?

Yes, I'm pretty sure that's the case.

If process A deletes the file, and Logstash "activates" 15 seconds later, then will it still have access to the old file?

Yes, but keep in mind that Logstash will periodically rediscover files (see the discover_interval option) and will reopen a file is its inode has changed.