# Aggregate - concatenate events

**URL:** <https://discuss.elastic.co/t/aggregate-concatenate-events/126808>\
**Category:** Logstash\
**Created:** [April 4, 2018, 7:02pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808 "2018-04-04T19:02:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![las](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@las](https://discuss.elastic.co/u/las)\
**Post date:** [April 4, 2018, 7:02pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/1 "2018-04-04T19:02:47Z")

</div>

12:12:12 processed T1  
data1 val1  
data2 val2  
dataId 1111  
data7 val7

I have configured filebeat to send multiline events like above to be sent to logstash.

In logstash, I am using grok and aggregate to extract the dataId value to combine multiple multiline events.

```
aggregate {
    task_id => "%{idValue}"
    code => "event.set('message', event.get('message') + map['message'])"
}

```

I don't want individual events to appear in elasticsearch - I tried event.cancel() but didn't seem to work.

I tried timeout and inactivity\_timeout configurations.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 4, 2018, 10:00pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/2 "2018-04-04T22:00:12Z")

</div>

Using the aggregate plugin to zip the messages back together is problematic because ordering is not strictly maintained; the multiline messages will need to be grouped on the Filebeat side before being transmitted to Logstash.

> If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.
> 
> -- [Filebeat Reference](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)

How is Filebeat configured? specifically, what are the values of the `multiline.*` fields?

---

<div class="post-metadata">

**Author:** ![las](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@las](https://discuss.elastic.co/u/las)\
**Post date:** [April 4, 2018, 10:15pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/3 "2018-04-04T22:15:36Z")

</div>

I have already taken care of filebeat and I'm getting multiple lines as 1 event. I'm trying to combine multiple of those based on an identifier. I'm able to group them but instead of getting only the aggregated one, I am getting 1, 1-2, 1-2-3 and so on.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 4, 2018, 10:30pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/4 "2018-04-04T22:30:53Z")

</div>

How do you know when an event is "done" and ready to be emitted? There are several generalised examples in [the aggregate filter plugin docs](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) that cover the various ways of configuring the plugin depending on what you expect.

Do you have example pipeline configuration that you can share?

---

<div class="post-metadata">

**Author:** ![las](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@las](https://discuss.elastic.co/u/las)\
**Post date:** [April 4, 2018, 11:38pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/5 "2018-04-04T23:38:48Z")

</div>

I have a conf file with input, filter and output.

In filter, I have grok and aggregate.

I am using timeout and inactivity\_timeout to aggregate events.

How do I avoid duplicates and get only the aggregated events?

---

<div class="post-metadata">

**Author:** ![las](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@las](https://discuss.elastic.co/u/las)\
**Post date:** [April 5, 2018, 3:39pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/6 "2018-04-05T15:39:11Z")

</div>

Do I need to use drop plugin if I dont want individual events to be sent to elasticsearch?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2018, 3:39pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808/7 "2018-05-03T15:39:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.