Drop Event older than 24 hours


(Nikhil Jaiswal) #1

Hi All,

I am using age filter to drop the event if time stamp is older than 24 hours, perhaps filter is not working properly because i am not able to see any output .

Any help regarding this will be appriciated .
please find my filter configuration below .

  filter {
if [message] =~ "^#" {
   drop {}
   }
	else if "DESKTOP-BT7BMJ0" in [host] {
		grok {
		  
		  #break_on_match => true
			
					match => [
					#"message", "%{GREEDYDATA:msg} Subject:.*Security ID:\s*%{NOTSPACE:security_id}\s*Account Name:\s*%{GREEDYDATA:account_name}\s*Account Domain:\s*%{NOTSPACE:account_domain}",
					"message", "%{GREEDYDATA}"						
					]
				remove_field => ["type"]
				remove_field => ["InsertionStrings"]
				  }

                
				
				  
			mutate {
				
				remove_field => ["keywords"]
				remove_field => ["beat"]
				remove_field => ["tags"]
				remove_field => ["user"]					
				remove_field => ["event_data"]	
				remove_field => ["provider_guid"]
				remove_field => ["process_id"]
				remove_field => ["thread_id"]
				 
				#rename => ["syslog_message" , "messages"]				
				
				
				add_field => ["Clientkey" , "SISAC1"]
				add_field => ["src_hostname" , "%{host}"]
				add_field => [ "src_ip", "%{host}" ]
				add_field => ["engine_id", "1518"]
				#add_field => [ 	"engine_log_id", "1"]
				rename => ["event_id" , "engine_log_id"]					
				#rename => ["host" , "src_hostname"]
				#rename => ["clienthost" , "client_IP"]
				#add_field => ["hostname","%{winname}"]
				remove_field => ["host"]	
				}

				age {}
if [@metadata][age] < 86400 {
		drop {}
		
				}
}

}

(Mehmet Sezgin) #2

Which version of logstash are you using?
age plugin isn't listed as input filter in documentation after 5.4 version.


(Magnus B├Ąck) #3

Your condition is backwards; you're dropping events that are younger than 24 hours.


(Nikhil Jaiswal) #4

i am using 5.6.X version of logstash.


(Christian Dahlqvist) #5

The age filter works based on the contents of the @timestamp field. If you have not parsed out a date from the event and used the date filter to populate the @timestamp field with this, dropping events based on the age filter will not work properly as @timestamp by default is set to the time Logstash received the event.

If you output the event to file or stdout instead of dropping it, you should be able to see this clearly.


(Nikhil Jaiswal) #6

thanks for writing .
Is it required to use date filter because logstash itself set the @timestamp for all events ?


(Christian Dahlqvist) #7

Yes, that is correct.


(Nikhil Jaiswal) #8

It worked , thanks a lot .


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.