Upload logs within a date range

[ManuMP]

We are using the following filter condition for uploading log entries. The filter will upload all log entries in the files. We would like to restrict the upload of log entries within a date range and not all log entries. How can I do that?

filter {

	if [type] == "connect300" {
		grok {
		  
		  match => { "message" => "\[%{NUMBER:LineNumber}]\[(?<logdate>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{TIME})\]-%{GREEDYDATA:level}-\[ThreadId = %{NUMBER:ThreadID}, %{GREEDYDATA:Module}]-%{GREEDYDATA:Message}\r" }
		}
		
		date {
			match => [ "logdate", "MM/dd/yyyy HH:mm:ss:SSS" ]
			target => "@timestamp"
		}
	}
}

[rugenl (Len Rugen)]

Not really pretty, but you could extract yyyy and mm from your logdate, then use logic like:

if [year] != "2019" and [mm] != "07" { drop {} }

These are character fields, so ranges would probably have to use regex. I don't do regex without study, but I think it would be something like [month] =~ "01|02|03", but don't use that without testing.

[Badger]

You could try this

    mutate {
        add_field => {
            "[@metadata][start]" => "2019-08-27T01:02:03.004Z"
            "[@metadata][end]" => "2019-08-27T22:02:03.004Z"
        }
    }
    date { match => [ "[@metadata][start]", "ISO8601" ] target => "[@metadata][start]" }
    date { match => [ "[@metadata][end]", "ISO8601" ] target => "[@metadata][end]" }
    ruby {
        code => '
            start = event.get("[@metadata][start]").to_f
            finis   = event.get("[@metadata][end]").to_f
            now   = event.get("@timestamp").to_f
            unless now < finis and now > start
                event.cancel
            end
        '
    }

[Ian_Boje]

There is an 'age' filter for logstash that might do what you want. I've got a filter to drop messages greater than a day old:

filter {
  age {}
  if [@metadata][age] > 86100 {
    drop {}
  }
}

There might be some concern that this filter might be deprecated, as I don't see it in the current documentation. We've got a few devices that periodically send log entries from weeks ago; I think the OS forgets the time for a split second. The age filter helped me.