Sincedb is not created

Elastic Stack Logstash
1

Hello all,

I just update my logstash server from 1.5.4 to 2.3.2. I have to update some config parameters and I am collect log from several file and indexing to elasticsearch.

My problem is when the logstash processs starts it doesn't create sincedb files and this cause data lose.

For instance, if i stop the deamon for a while and i start it again few minutes later, logstash doesn't continue in the line where the process stop, it starts with lastest lines.

This is my logstash configuration:

input {
file {
path => "/var/log/varnish/varnishncsa.log"
type => "varnish"
sincedb_write_interval => "5"
sincedb_path => "/opt/logstash/logs/.since_varnishncsa.log"
}

file {
    path => "/var/log/varnish/recovery.log"
    type => "varnish"
    sincedb_write_interval => "5"
    sincedb_path => "/opt/logstash/logs/.since_recovery.log"
}

file {
    path => "/var/log/nginx/access.log"
    type => "https"
    sincedb_write_interval => "5"
    sincedb_path => "/opt/logstash/logs/.since_access_nginx.log"
}

}

filter {
if [type] == "varnish" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => { "message" => "(%{IP:clientip}|-) (%{USER:ident}|-) (%{USER:auth}|-) [%{HTTPDATE:timestamp}] "%{WORD:method} http://%{HOSTNAME:vhost}%{URIPATHPARAM:request}([%{WORD:garbage}])? HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:agent} (%{WORD:cached}|-) %{QS:content_type} %{QS:node} %{NUMBER:response_time} (%{QS:clientid}|-) (%{QS:referer}|-)"}
}

        date {
                    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
                    add_tag => [ "dated" ]
        }

}
if [type] == "https" {
        grok {
                    patterns_dir => "/opt/logstash/patterns"
                    match => { "message" => "(%{IP:clientip}|-) (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] %{HOSTNAME:vhost} \"%{WORD:method} %{URIPATHPARAM:request}(\[%{WORD:garbage}\])? HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent} (%{QS:clientid}|-)"}
        }

        date {
                    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
                    add_tag => ["dated"]
        }
}

}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
flush_size => "5000"
idle_flush_time => "1"
#index => "logstash-%{index_date}"
index => "logstash-%{+YYYY.MM.dd.HH}"
}
}

Any clue???

2

What user does Logstash run as? What are the file permissions of /opt/logstash/logs and /opt/logstash/logs/.since_recovery.log? Are there any clues in the Logstash log is you crank up the log level with --verbose or --debug?

3

Hey Magnus, thanks for your reply,

Logstash is running as root user, so is not a privileges problem, in fact its so weird because i have 3 input file (/var/log/varnish/varnishncsa.log, /var/log/nginx/access.log and /var/log/varnish/recovery.log) and only I hae problem with the first one (the other are created and updated right).

Curiously, /var/log/varnish/varnishncsa.log is the most updated file that is because it's so odd this behavior.

I run logstash with --verbose options and all i can see was this error but I don't what that means:

{:timestamp=>"2016-06-13T20:00:54.898000+0200", :message=>"Ruby exception occurred: can't dup NilClass", :level=>:error}

Jorge

4

There's no mention of any of the sincedb files in the --debug output? Is there any more context to the "can't dup NilClass" error?

5

yes i can see this when i run logstash in debug mode:

{:timestamp=>"2016-06-14T09:09:42.537000+0200", :message=>"writing sincedb (delta since last write = 5)", :level=>:debug, :file=>"filewatch/observing_tail.rb", :line=>"102", :method=>"observe_read_file"}

however the file /opt/logstash/logs/.since_varnishncsa.log is not created.

i have seen a very interesting thing. Sometimes, when I have less events to send (no peak time), the file is created and updated frequently. However in peak time the some time the sincedb file is not created and when is created is not updated frequently.

In logstash 1.5 this configuration was working fine. I really don't know what is happend.

6

I got the same problem with Logstash 2.3.2. Since-DB file won't be updated if there are a lot of events...