Run a filter that checks for values in previous records

[Tejas_Ghutukade]

Hi,

i am passing a csv file to lagstach with records of timestamp and processing time.
i want to create a fliter that checks if the current timestamp lies within the timestamp and processing time of any of the previous records and if condition true want to add a field with value 1 .

is it possible to do this ?

Thanks

[Rory]

You should take a look at the logstash aggregate filter: https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

One thing I would suggest is to sort the csv file by timestamp...

[Tejas_Ghutukade]

@Rory

v file by ti

yes its already sorted.I'll have a look at the aggregate filter.

Thanks

[Tejas_Ghutukade]

@Rory
I tried aggregate filter. here's my filter code

aggregate {
	task_id => "%{TRANSACTIONIDGLOBAL}"
	code => " 	
	map['tr_start'] ||= 0;
	map['tr_end'] ||= 0;
	event.set('cc' , event.get('starttime') < map['tr_end']);			
	map['tr_start'] = event.get('starttime');
	map['tr_end'] = event.get('endtime');
	"
	push_map_as_event_on_timeout => true
	timeout => 120
}	

so here m trying to check if for the current record the starttime is whether less than the previous record endtime.
but in output its checking the start time with the endtime of the current record itself.
at the comparision level istead of checking with the previous record its checking with itself.

Thanks