Drop Event older than 24 hours

Elastic Stack Logstash
1

Hi All,

I am using age filter to drop the event if time stamp is older than 24 hours, perhaps filter is not working properly because i am not able to see any output .

Any help regarding this will be appriciated .
please find my filter configuration below .

  filter {
if [message] =~ "^#" {
   drop {}
   }
	else if "DESKTOP-BT7BMJ0" in [host] {
		grok {
		  
		  #break_on_match => true
			
					match => [
					#"message", "%{GREEDYDATA:msg} Subject:.*Security ID:\s*%{NOTSPACE:security_id}\s*Account Name:\s*%{GREEDYDATA:account_name}\s*Account Domain:\s*%{NOTSPACE:account_domain}",
					"message", "%{GREEDYDATA}"						
					]
				remove_field => ["type"]
				remove_field => ["InsertionStrings"]
				  }

                
				
				  
			mutate {
				
				remove_field => ["keywords"]
				remove_field => ["beat"]
				remove_field => ["tags"]
				remove_field => ["user"]					
				remove_field => ["event_data"]	
				remove_field => ["provider_guid"]
				remove_field => ["process_id"]
				remove_field => ["thread_id"]
				 
				#rename => ["syslog_message" , "messages"]				
				
				
				add_field => ["Clientkey" , "SISAC1"]
				add_field => ["src_hostname" , "%{host}"]
				add_field => [ "src_ip", "%{host}" ]
				add_field => ["engine_id", "1518"]
				#add_field => [ 	"engine_log_id", "1"]
				rename => ["event_id" , "engine_log_id"]					
				#rename => ["host" , "src_hostname"]
				#rename => ["clienthost" , "client_IP"]
				#add_field => ["hostname","%{winname}"]
				remove_field => ["host"]	
				}

				age {}
if [@metadata][age] < 86400 {
		drop {}
		
				}
}

}
2

Which version of logstash are you using?
age plugin isn't listed as input filter in documentation after 5.4 version.

3

Your condition is backwards; you're dropping events that are younger than 24 hours.

4

i am using 5.6.X version of logstash.

1 Like
5

The age filter works based on the contents of the @timestamp field. If you have not parsed out a date from the event and used the date filter to populate the @timestamp field with this, dropping events based on the age filter will not work properly as @timestamp by default is set to the time Logstash received the event.

If you output the event to file or stdout instead of dropping it, you should be able to see this clearly.

6

thanks for writing .
Is it required to use date filter because logstash itself set the @timestamp for all events ?

7

Yes, that is correct.

8

It worked , thanks a lot .

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.