How does sincedb work for a Logstash reading multiple files from a single directory?

[amruth]

Hi,

I am wondering how sincedb file works for multiple files in a single directory. When I start Logstash and once it process all the files sincedb file has only one line(probably just the information of a single file). But when I stop the Logstash, I see the sincedb file has the information about all files. I have a config file like,

file{
		path=>"C:/Documents/*.out"
		start_position=> "beginning"
		sincedb_path => "C:/logstash-5.3.1/logstash-5.3.1/data/plugins/inputs/file/sincedb.log"
		type=>"failure"
}

Please shed some light on this. Thank you.

[amruth]

Can someone please help on this?

[guyboertje]

Is the documentation not enough? https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files

Please tell us how we can improve the docs.

The file input holds a map of inode -> current-position in memory. Every sincedb_write_interval the map is persisted to disk. On Logstash shut down, its also persisted. Some times the file input thread is blocked by the queue and it doesn't get round to persisting the map.

[amruth]

Hi Guy,

For my above mentioned example, I am using a single sincedb file for multiple input files. So how would the sincedb file look? Would it have multiple entries(information about all the input files)?

Thanks

[guyboertje]

Yes.

Something like this. 5 files were read, first number is the inode and the last number is the bytes read.

20670166 1 4 1549041625
16400297 1 4 107043
22063260 1 4 91
14993529 1 4 281500
13377766 1 4 281500

[amruth]

Makes sense. Thank you