How does Logstash behave when Input file content is deleted?

Elastic Stack Logstash
1

Hi,

I have a Logstash setup where it reads file(Linux server logs) as an input. But for every 5 days, content in that file will be deleted and new logs will be placed in the input file. Will Logstash be reading lines from beginning? I am confused here. Can someone please explain?

Thanks

2

Exactly what happens to the file? Is it file deleted? Or truncated? Or something else?

3

Sorry for the confusion. File name is server.log and for every 5 days it becomes server-x.log and a new file is created with the name server.log. Process repeats for every 5 days.

4

See https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_file_rotation_2.

5

I've read that, but I am not clear at that. Can you please explain taking my current scenario as an example?
And this is how my input config looks like,

input{

        file{
                path=>"/opt/logs/server.log"
                start_position=> "beginning"
                sincedb_path => "/opt/elk/logstash/sincedb/success_sincedb.log"
                type=>"success"

                codec => multiline {
                        pattern => "^success"
                        negate => true
                        what => previous
                        #auto_flush_interval => 10
                        max_lines => 100000
                        max_bytes => "1000 MiB"
                }
        }
}
6

Hi Magnus, can you please explain?

7

You're not saying what part is unclear so I'm not sure what to explain. When the file is renamed Logstash will discover this and start reading the newly created file instead.

8

When it starts reading the newly created file(since in the input I declared "path=>"/opt/logs/server.log""), how would sincedb file change?

The reason I am asking this is - when it's reading server.log, Logstash stores in the sincedb file that it has read till 100th line and on the fifth day when server.log is renamed to server1.log and a new file server.log is created(with the same name), how will Logstash behave here? Will it read from 1st line or from 101 line?

Or is it like sincedb file gets updated only when Logstash is shut down? If that's the case then what happens when Logstash shutting down and renaming the file happens at the same time?

9

how would sincedb file change?

A new sincedb entry will be created since the new file has a new inode number. Logstash tracks file by inode numbers, not names. See the documentation.

The reason I am asking this is - when it's reading server.log, Logstash stores in the sincedb file that it has read till 100th line

It stores a byte offset, not a line number. Again, see the documentation.

and on the fifth day when server.log is renamed to server1.log and a new file server.log is created(with the same name), how will Logstash behave here? Will it read from 1st line or from 101 line?

The file will be read from the top.

10

Thanks a lot for the detailed explanation.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.