Make event wait

Elastic Stack Logstash
1

Hi,

Can I make that if I get a log that parameter X = 'a' that Logstash will not push it straight to ES but wait for 3 minutes?

2

The only way to do that would be to pause the entire pipeline, maybe with a ruby hack (note: I call it a hack because it is, Logstash is not designed for this)

3

Understood, the reason I am trying this hack is b/c "push_map_as_event_on_timeout => true" doesn't work for me...
at the end of timeout I dont have a new log containing all the map

4

Perhaps if you provided more detail on the problem others can help?

5

Yes, well I am able to aggregate that my last log has all the terms, but I can't know who will be my last log, therefore I did "push_map_as_event_on_timeout => true" (I kept all config that is not repetitive since maybe some part that I didn't think of destroys the push map.

This is suppose to print out a new log after 120 seconds isn't it? (for me it is not working)

Here is my LS config:

input {
 ...
}

filter {
  json{
        source => "message"
  }

  aggregate {
		task_id => "%{transactionId}"
		code => "
			
			if (map['authCBGot'])
				event.set('authCBGot', (map['authCBGot']))
			else 
				if (event.get('authCBGot').eql? '0')
					# do nothing
				else
					map['authCBGot'] = event.get('authCBGot')
				end
			end
			... (many more as the "authCBGot")
			"
		push_map_as_event_on_timeout => true
		timeout_task_id_field => "transactionId"
		timeout => 120
		timeout_code => "event.set('cdrType', 'aggregated')"
    
   }
   
  if [ASR]{
	  mutate {
		convert => { "ASR" => "float" }
	  }
  }
  
  date {
    match => [ "eventTimestamp", "UNIX_MS" ]
    target => "eventTimestamp"
  }
  ...(many more Date filters)
  
 (some ruby)
  if [medGotAck_MT] and [MedGotMsgFromApi] {
    ruby {
      init => "require 'time'"
      code => "
					medGotAck_MT = Time.iso8601(event.get('medGotAck_MT').to_s).to_i;
					MedGotMsgFromApi = Time.iso8601(event.get('MedGotMsgFromApi').to_s).to_i;
					event.set('delay' , medGotAck_MT - MedGotMsgFromApi);
					if (event.get('cdrType').eql? 'dlr')
						if (event.get('MedGotMsgFromApi').eql? '0')
							event.set('delay' , 120);
						else
							event.set('delay' , medGotAck_MT - MedGotMsgFromApi);
						end
					else
						event.set('delay' , medGotAck_MT - MedGotMsgFromApi);
					end
					event.set('epoch_received_at_in_seconds' , medGotAck_MT);
					event.set('epoch_timestamp_in_seconds' , MedGotMsgFromApi);
				"
			
    }
  }
  
  
	 
}

output {
    elasticsearch {
      hosts => ["192.168.1.116:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
  }
  


}
6

@warkolm should I ask this also as a new topic?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.