# Combining the logs in logstash

**URL:** <https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682>\
**Category:** Logstash\
**Created:** [July 16, 2019, 8:39am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682 "2019-07-16T08:39:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 16, 2019, 8:39am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/1 "2019-07-16T08:39:46Z")

</div>

Hi,

I need to combine the two log lines using the message id i tried aggregate filter plugin but i cant achieve it.Please guide

```
Jun 5 19:24:18 gpigw11 postfix/qmgr[37531]: 3077F92E45: from=< ****** >, size=2801, nrcpt=1 (queue active)

```

Jun 5 19:24:18 gpigw11 postfix/lmtp[99429]: 3077F92E45: to=\<\*\*\*\*\*\*\>, delay=0.2, delays=0.01/0/0.05/0.14, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 16, 2019, 12:55pm UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/2 "2019-07-16T12:55:16Z")

</div>

The following would do it. Note that it drops the original events, so the only fields that will be on the final event are the ones that were stored in the map.

Also, the -\> in the dissect filter has the effect of compacting the double space down to a single space, so you do not need a second pattern in the date filter to hand the double space case for the first 9 days of the month.

You may need a longer timeout than 10 seconds.

```
    dissect { mapping => { "message" => "%{[@metadata][ts]->} %{+[@metadata][ts]} %{+[@metadata][ts]} %{theHost} %{}/%{}[%{}]: %{task}: %{[@metadata][kvStuff]}" } }
    date { match => ["[@metadata][ts]", "MMM dd HH:mm:ss" ] }
    kv { source => "[@metadata][kvStuff]" target => "kvData" field_split => ", " }
    aggregate {
        task_id => "%{task}"
        push_map_as_event_on_timeout => true
        inactivity_timeout => 10
        timeout_task_id_field => "correlationId"
        code => '
            map["kvData"] ||= {}
            map["kvData"] = map["kvData"].merge(event.get("kvData"))
            map["theHost"] = event.get("theHost")
            map["@timestamp"] = event.get("@timestamp")
            event.cancel
        '
    }
```

---

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 17, 2019, 5:06am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/3 "2019-07-17T05:06:29Z")

</div>

Thanks a lot will check and let u know..

---

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 17, 2019, 6:46am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/4 "2019-07-17T06:46:15Z")

</div>

> [@Badger](#):
>
> dissect { mapping =\> { "message" =\> "%{[@metadata][ts]-\>} %{+[@metadata][ts]} %{+[@metadata][ts]} %{theHost} %{}/%{}[%{}]: %{task}: %{[@metadata][kvStuff]}" } } date { match =\> ["[@metadata][ts]", "MMM dd HH:mm:ss" ] } kv { source =\> "[@metadata][kvStuff]" target =\> "kvData" field\_split =\> ", " } aggregate { task\_id =\> "%{task}" push\_map\_as\_event\_on\_timeout =\> true inactivity\_timeout =\> 10 timeout\_task\_id\_field =\> "correlationId" code =\> ' map["kvData"] ||= {} map["kvData"] = map["kvData"].merge(event.get("kvData")) map["theHost"] = event.get("theHost") map["@timestamp"] = event.get("@timestamp") event.cancel ' }

hi can u please explain how this works...i have seen this dissect for the first time

---

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 17, 2019, 7:01am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/5 "2019-07-17T07:01:54Z")

</div>

and i m getting this warning [2019-07-17T12:18:31,365][WARN][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=\>"message", "event"=\>{"host"=\>"localhost.localdomain", "message"=\>"", "tags"=\>["\_dissectfailure"], "@version"=\>"1", "@timestamp"=\>2019-07-17T06:48:31.261Z}} and this is also getting stored in elasticindex and displaying in kibana...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 17, 2019, 2:15pm UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/6 "2019-07-17T14:15:42Z")

</div>

If you want to understand dissect better I would start with the [blog post](https://www.elastic.co/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs) that introduced it and the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html).

The warning occurs because the message field is an empty string ("message"=\>""), so it does not match the dissect mapping. The documentation explains that [conditional processing](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html#_conditional_processing) may be needed to avoid these warnings.

---

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 18, 2019, 4:44am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/7 "2019-07-18T04:44:53Z")

</div>

Thanks a lot....Will read the documents

---

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 18, 2019, 10:58am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/8 "2019-07-18T10:58:04Z")

</div>

i have read the docs and cleared the warnings....it is storing all the combined logs in kvData format which is ok...there is a small problem in storing the fields if u see my logs the last field status has some line in the brackets for e.g status=sent (250 2.1.5 Delivery OK) while storing the value in kvData it is storing it as kvData.status=sent but the lines in bracket are not included this is because we have given the field\_split =\> ", " so how to store the whole status field...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 18, 2019, 11:32am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/9 "2019-07-18T11:32:56Z")

</div>

> [@Badger](#):
>
> kv { source =\> "[@metadata][kvStuff]" target =\> "kvData" field\_split =\> ", " }

Remove the space from field\_split and add trim\_key =\> " "

---

<div class="post-metadata">

**Author:** ![vinu89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinu89/32/45939_2.png) [@vinu89](https://discuss.elastic.co/u/vinu89)\
**Post date:** [July 19, 2019, 4:34am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/10 "2019-07-19T04:34:01Z")

</div>

ok thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 16, 2019, 4:34am UTC](https://discuss.elastic.co/t/combining-the-logs-in-logstash/190682/11 "2019-08-16T04:34:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.