Could not load : can't convert nil into String?

Elastic Stack Logstash
1

I want to read logs only after a particular date. So my approach is to drop all the events previous to that date. I try to achieve it like this:

So I am dropping all logs before June 1, 2015:

Logstash config file:

input {
         file{
                path => [
                         "/var/log/rsyslog/**/*.log"
                ]
        }
}

filter {

        grok {

        match => ["path", "/var/log/rsyslog/(?<server>[^/]+)/%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:month_day}/(?<logtype>.*).log"]

        }

        if [year] < "2015" and [month] < "6" and [month_day] < "1" {

                drop { }

        }

My logstash.err file keeps printing this:

Could not load : can't convert nil into String

Any idea why ??

2

Have you double-checked that the year, month, and monthday fields are defined?

You should convert the fields to integers with %{YEAR:year:int} etc so that you can say

if [year] < 2015 and [month] < 6 and [month_day] < 1 {

instead. As it stands you'll drop all messages from October, November, and December since "10" < "6" is true (but 10 < 6 is false).

3

Actually I am getting this error:

{:timestamp=>"2015-06-03T16:29:26.478000+0530", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"%{timestamp} 2015", :exception=>java.lang.IllegalArgumentException: Invalid format: "%{timestamp} 2015", :level=>:warn}

and my timestamp field as taken from log does not has a year so I add year field to timestamp:

        mutate {
                replace => ["timestamp", "%{timestamp} %{year}"]
        }

date {
                "locale" => "en"
                match => ["timestamp", "MMM  d HH:mm:ss YYYY", "MMM dd HH:mm:ss YYYY", "ISO8601"]
                target => "@timestamp"
        }
4

The timestamp field hasn't been set.

5

The timestamp filed as read is replaced with year added in that.
Where is i going wrong ?

6

The evidence suggests that the timestamp isn't set when you attempt to append the contents of the year field. That's why the value ends up as "%{timestamp} 2015", because non-existent fields referenced with the %{varname} notation are left untouched. When is timestamp set the first time?

7

timestamp is set using grok:

grok{
                match => ["message", "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST} %{SYSLOGPROG}: %{GREEDYDATA:rsyslog-message}"]
        }

and when I check my logstash.stdout I can see the output:

{
          "message" => "Jun  3 12:02:28 server1 snmpd[6234]: some message",
         "@version" => "1",
       "@timestamp" => "2015-06-03T06:32:28.000Z",
             "path" => "/var/log/rsyslog/server1/2015-06-03/snmpd.log",
           "server" => "server1",
             "year" => 2015,
            "month" => 6,
        "month_day" => 3,
          "logtype" => "snmpd",
        "timestamp" => "Jun  3 12:02:28 2015",
          "program" => "snmpd",
         "snmpd-id" => "6234",
    "snmpd.message" => "some message"
}
8

And this grok filter comes prior to the mutate filter? Please show your full configuration.

9 
filter {

        grok {

        match => ["path", "/var/log/rsyslog/(?<server>[^/]+)/%{YEAR:year:int}-%{MONTHNUM:month:int}-%{MONTHDAY:month_day:int}/(?<logtype>.*).log"]

        }

        if [year] < 2015{
                drop {}
        }else if [year] == 2015{

                if [month] < 6 {
                        drop {}
                }

        }

        if [logtype] == "snmpd" {

        grok{
                match => ["message", "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST} %{SYSLOGPROG}: %{GREEDYDATA:rsyslog-message}"]
        }

        }

 mutate {
                replace => ["timestamp", "%{timestamp} %{year}"]
        }

        date {
                "locale" => "en"
                match => ["timestamp", "MMM  d HH:mm:ss YYYY", "MMM dd HH:mm:ss YYYY", "ISO8601"]
                target => "@timestamp"
        }
10

The grok filter that populates timestamp only executes if logtype is "snmpd" but the date filter isn't wrapped by the same kind of conditional.

11

Do you have any idea about this: Logstash close file handles?