[AGGREGATE] - aggregate is working but single rows are sent to elastic too

Elastic Stack Logstash
1

Hi, I'm trying to get some exchange log to elastic using Logstash, I've tried some configuration and found something that is working but if I remove the event.cancel() in all the if conditions all rows are sent to elasticsearch and not only the aggregate.
Here is my logstash configuration

grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:event_time},%{WORD:server_name}\\%{GREEDYDATA:connector_id},%{BASE16FLOAT:sessionID},%{INT:seq_number},%{IPV4:local_IP}:%{POSINT:local_port},%{IPV4:remote_IP}:%{POSINT:remote_port},%{DATA:event},%{GREEDYDATA:conn_mess}"}
	}

	if [event] == "+" {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_mess'] = event.get('conn_mess');
					event.cancel()"
			map_action => "create"
		}
		
	}
	if [event] == ">" {
		aggregate {
			task_id => "%{sessionID}"
			
			code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
			event.cancel()"
		}
		
	}
	if [event] == "<" {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
			event.cancel()"
		}
		
	}
	if [event] == "*" {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
			event.cancel()"
			
		}
		
	}		
	if [event] == "-" {
		aggregate {
			task_id => "%{sessionID}"
			code => "event.set('event_time', event.get('event_time'))
					event.set('server_name', event.get('server_name'))
					event.set('connector_id', event.get('connector_id'))
					event.set('sessionID', event.get('sessionID'))
					event.set('seq_number', event.get('seq_number'))
					event.set('local_IP', event.get('local_IP'))
					event.set('local_port', event.get('local_port'))
					event.set('remote_IP', event.get('remote_IP'))
					event.set('remote_port', event.get('remote_port'))
					event.set('event',event.get('event'))
					event.set('conn_user',map['conn_user'])
					event.set('conn_mess', map['conn_mess'])"
			
			push_previous_map_as_event => true
			end_of_task => true
			timeout => 120
			add_tag => ['fine']
		}
		
	}

My final goal is to add this code

		if "User Name" in [conn_mess] {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_user'] ||= event.get('conn_mess'); map['conn_user'] << event.get('conn_mess');
			event.cancel()"
		}
	}

in order to have a new field with the "conn_mess" string only for the row matching the if statement

2

That's exactly what is expected to happen.

1 Like
3

Ok, sò how can I map the row "conn_mess" in a new map element and add It in the aggregate element? Actually I have the "conn_user" field always as nil.
Thx!

4

None of these

event.set('connector_id', event.get('connector_id'))

make any sense. It is fetching a field from the event and putting it back into the event. It is a no-op.

Can you show some sample data and show what you want the final event to look like?

5

This was my last test, so I will remove those lines on the last if and I leave there only the set with the map?
Tomorrow I can send some rows

6

I've found a solution, my problem was in the "event.cancel" now i've nested two if statement and all is working as expected

if [event] == "*" {
		if "User Name" in [conn_mess] {
			aggregate {
				task_id => "%{sessionID}"
				code => "map['conn_user'] = event.get('conn_mess')[12..(event.get('conn_mess').length)];
						map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
				event.cancel()"
				}
		}else{
			aggregate {
				task_id => "%{sessionID}"
				code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
				event.cancel()"
				
			}
		}
	}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.