# Logstash Aggregate plugin not working **URL:** \ **Category:** Logstash\ **Created:** [September 29, 2020, 2:14pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371 "2020-09-29T14:14:26Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rfferrao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rfferrao/32/76344_2.png) [@rfferrao](https://discuss.elastic.co/u/rfferrao)\ **Post date:** [September 29, 2020, 2:14pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/1 "2020-09-29T14:14:26Z")
Trying to implement aggregate plugin based on the documentation, but I'm having problems. Here's how I've configured it: ```auto aggregate { task_id => "%{tunnelid}" code => " map['user'] = event.get('user') map['group'] = event.get('group') map['remip'] = event.get('remip') map['logs'] = [] map['logs'] << { 'action' => event.get('action'), 'datetime' => event.get('datetime') } event.cancel() " map_action => "create" timeout => 30 timeout_code => "event.set('state', 'timeout')" push_map_as_event_on_timeout => true } ``` I've set `pipeline.workers` to 1 in the `pipelines.yml` configuration file, as shown below: ```auto - pipeline.id: main path.config: "/etc/logstash/conf.d/*.conf" pipeline.workers: 1 ``` The output in Elasticsearch however always ends up unformatted like this: ```auto [{'_index': 'sslvpnteste-2020.09.29', '_type': '_doc', '_id': 'fysk2nQBwfAlkZKXBBW9', '_score': 1.0, '_source': {'user': 'someuser', 'group': 'AcessoVpn', 'remip': '192.168.0.1', 'action': 'tunnel-down', 'datetime': '2020-09-29T10:51:47-0300', 'message': '<190>Sep 29 10:53:46 192.168.0.1 action="tunnel-down" tunnelid=862562769 remip=192.168.0.1 user="someuser" group="AcessoVpn"', 'log_type': 'sslvpnteste', 'tunnelid': '862562769'}}] ``` Where am I going wrong? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 29, 2020, 3:43pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/2 "2020-09-29T15:43:55Z")
> [@rfferrao](#): > > Where am I going wrong? I do not see anything wrong. The \_source has a bunch of fields in it. Do they not show up if you refresh the index in kibana? You might want to look at the timeout\_task\_id\_field option on the aggregate filter if you want to keep the tunnelid. ---
**Author:** ![rfferrao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rfferrao/32/76344_2.png) [@rfferrao](https://discuss.elastic.co/u/rfferrao)\ **Post date:** [September 29, 2020, 3:47pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/3 "2020-09-29T15:47:18Z")
@Badger According to my `code` I should have an array field called `logs` wherein `action` and `datetime` should be stored as a hash, but this map doesn't appear to be applied. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 29, 2020, 3:52pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/4 "2020-09-29T15:52:48Z")
> [@rfferrao](#): > > According to my `code` I should have an array field called `logs` where `action` and `datetime` should be stored as a hash Indeed. Note that the event from elasticsearch has a message field, but you never add [message] to the map. That is not an aggregated event, it is the event you are trying to aggregate. How that could get to elasticsearch when you have event.cancel in the aggregate code option is beyond me. Maybe you are not running the configuration you think you are running. ---
**Author:** ![rfferrao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rfferrao/32/76344_2.png) [@rfferrao](https://discuss.elastic.co/u/rfferrao)\ **Post date:** [September 29, 2020, 4:00pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/5 "2020-09-29T16:00:29Z")
> [@Badger](#): > > but you never add [message] to the map. That is not an aggregated event, it is the event you are trying to aggregate. Sorry, but could you explain this part in more detail? I'm not sure I understood what you meant by this. Also, I've already tried running the same `code` without `event.cancel`, but there's no difference in the output - also, at least to me, the documentation is not very clear on what it actually does; I was assuming that the original (non-aggregated) event was thrown away and the aggregated event was preserved. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 29, 2020, 5:41pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/6 "2020-09-29T17:41:44Z")
> [@rfferrao](#): > > I was assuming that the original (non-aggregated) event was thrown away and the aggregated event was preserved. That is exactly what it does. If it is not doing it then event.cancel is not being called. Remove ``` map_action => "create" ``` The filter will [return before executing the code](https://github.com/logstash-plugins/logstash-filter-aggregate/blob/8603b168d93c43a4bbc72ca92a633d6c5276439e/lib/logstash/filters/aggregate.rb#L212) if that is present. ---
**Author:** ![rfferrao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rfferrao/32/76344_2.png) [@rfferrao](https://discuss.elastic.co/u/rfferrao)\ **Post date:** [September 29, 2020, 5:55pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/7 "2020-09-29T17:55:39Z")
Can't believe it was that simple! Thank you very much once again, @Badger. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 27, 2020, 5:55pm UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin-not-working/250371/8 "2020-10-27T17:55:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.