Sincedb files not created

Elastic Stack Logstash
1

Hi,
The following is my sample Logstash configuration for file input:

file {
                path => ".../logs/sample.log"
                start_position => "beginning"
                type => "sample"
                sincedb_path => "$HOME/logstash/sincedb/"
}

I see that no sincedb files are being created.
I also tried the following:

# without slash at the end of the path
file {
           ...                
           sincedb_path => "$HOME/logstash/sincedb"
}

# directly providing wildcard for sincedb files
file {
           ...                
           sincedb_path => "$HOME/logstash/sincedb/.sincedb*"
}

None of them seems to be working.
What is the correct way to give sincedb path?
Need help on this.

2

I wouldn't assume that $HOME is expanded here. Also, I suspect sincedb_path should point to a file rather than a directory. Increasing the log verbosity with --verbose or even --debug should give more insights into what's going on.

Any particular reason why you want to set sincedb_path? The default value is fine in most circumstances.

3

@magnusbaeck

I shall check with --debug information.

As I understand, whenever we bounce the logstash, if I don't set sincedb_path:

  • if start_position => beginning : logs would be read again
  • else start_position => end : logs written during the bouncing would be missed

So, I would require sincedb_path to avoid the above two scenarios.

Please correct me if I am wrong.

4

Logstash will generate a sincedb path based on the filename pattern so regardless it'll always remember where it left off as long as you don't change the filename pattern. So yes, there are occasions where setting sincedb_path makes sense but I'd say that people are overusing it.

5

@magnusbaeck

Are you saying that sincedb files are being saved in some path based on filename pattern? But we might not have write permissions in those paths. I haven't seen sincedb files anywhere yet.
Can you please elaborate. Because, its important not to loose any data after bouncing logstash.

6

As documented, the sincedb_path default causes files to be written to $HOME, i.e. the home directory of whatever user that Logstash runs as (typically logstash). It's a reasonable assumption that those files will be writable to Logstash. See below for the code that sets sincedb_path dynamically when the user hasn't set it.

And again, if you increase the logging verbosity Logstash will tell you more about the location of the sincedb files.

1 Like