# \[AGGREGATE\] - aggregate is working but single rows are sent to elastic too

**URL:** <https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880>\
**Category:** Logstash\
**Created:** [February 18, 2019, 5:12pm UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880 "2019-02-18T17:12:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![stestaz](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@stestaz](https://discuss.elastic.co/u/stestaz)\
**Post date:** [February 18, 2019, 5:12pm UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/1 "2019-02-18T17:12:18Z")

</div>

Hi, I'm trying to get some exchange log to elastic using Logstash, I've tried some configuration and found something that is working but if I remove the event.cancel() in all the if conditions all rows are sent to elasticsearch and not only the aggregate.  
Here is my logstash configuration

```
grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:event_time},%{WORD:server_name}\\%{GREEDYDATA:connector_id},%{BASE16FLOAT:sessionID},%{INT:seq_number},%{IPV4:local_IP}:%{POSINT:local_port},%{IPV4:remote_IP}:%{POSINT:remote_port},%{DATA:event},%{GREEDYDATA:conn_mess}"}
	}

	if [event] == "+" {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_mess'] = event.get('conn_mess');
					event.cancel()"
			map_action => "create"
		}
		
	}
	if [event] == ">" {
		aggregate {
			task_id => "%{sessionID}"
			
			code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
			event.cancel()"
		}
		
	}
	if [event] == "<" {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
			event.cancel()"
		}
		
	}
	if [event] == "*" {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
			event.cancel()"
			
		}
		
	}		
	if [event] == "-" {
		aggregate {
			task_id => "%{sessionID}"
			code => "event.set('event_time', event.get('event_time'))
					event.set('server_name', event.get('server_name'))
					event.set('connector_id', event.get('connector_id'))
					event.set('sessionID', event.get('sessionID'))
					event.set('seq_number', event.get('seq_number'))
					event.set('local_IP', event.get('local_IP'))
					event.set('local_port', event.get('local_port'))
					event.set('remote_IP', event.get('remote_IP'))
					event.set('remote_port', event.get('remote_port'))
					event.set('event',event.get('event'))
					event.set('conn_user',map['conn_user'])
					event.set('conn_mess', map['conn_mess'])"
			
			push_previous_map_as_event => true
			end_of_task => true
			timeout => 120
			add_tag => ['fine']
		}
		
	}

```

My final goal is to add this code

```
		if "User Name" in [conn_mess] {
		aggregate {
			task_id => "%{sessionID}"
			code => "map['conn_user'] ||= event.get('conn_mess'); map['conn_user'] << event.get('conn_mess');
			event.cancel()"
		}
	}

```

in order to have a new field with the "conn\_mess" string only for the row matching the if statement

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2019, 6:19pm UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/2 "2019-02-18T18:19:51Z")

</div>

> [@stestaz](#):
>
> if I remove the event.cancel() in all the if conditions all rows are sent to elasticsearch and not only the aggregate.

That's exactly what is expected to happen.

---

<div class="post-metadata">

**Author:** ![stestaz](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@stestaz](https://discuss.elastic.co/u/stestaz)\
**Post date:** [February 18, 2019, 8:17pm UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/3 "2019-02-18T20:17:20Z")

</div>

Ok, sò how can I map the row "conn\_mess" in a new map element and add It in the aggregate element? Actually I have the "conn\_user" field always as nil.  
Thx!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2019, 9:16pm UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/4 "2019-02-18T21:16:38Z")

</div>

None of these

```
event.set('connector_id', event.get('connector_id'))

```

make any sense. It is fetching a field from the event and putting it back into the event. It is a no-op.

Can you show some sample data and show what you want the final event to look like?

---

<div class="post-metadata">

**Author:** ![stestaz](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@stestaz](https://discuss.elastic.co/u/stestaz)\
**Post date:** [February 18, 2019, 10:08pm UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/5 "2019-02-18T22:08:11Z")

</div>

This was my last test, so I will remove those lines on the last if and I leave there only the set with the map?  
Tomorrow I can send some rows

---

<div class="post-metadata">

**Author:** ![stestaz](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@stestaz](https://discuss.elastic.co/u/stestaz)\
**Post date:** [February 19, 2019, 8:40am UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/6 "2019-02-19T08:40:56Z")

</div>

I've found a solution, my problem was in the "event.cancel" now i've nested two if statement and all is working as expected

```
if [event] == "*" {
		if "User Name" in [conn_mess] {
			aggregate {
				task_id => "%{sessionID}"
				code => "map['conn_user'] = event.get('conn_mess')[12..(event.get('conn_mess').length)];
						map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
				event.cancel()"
				}
		}else{
			aggregate {
				task_id => "%{sessionID}"
				code => "map['conn_mess'] ||= event.get('conn_mess'); map['conn_mess'] << event.get('conn_mess');
				event.cancel()"
				
			}
		}
	}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 19, 2019, 8:40am UTC](https://discuss.elastic.co/t/aggregate-aggregate-is-working-but-single-rows-are-sent-to-elastic-too/168880/7 "2019-03-19T08:40:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.