# Combine Information from different log lines into one event

**URL:** <https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518>\
**Category:** Logstash\
**Tags:** elastic-stack-alerting\
**Created:** [November 18, 2021, 2:43am UTC](https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518 "2021-11-18T02:43:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mangeshmj1992](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\
**Post date:** [November 18, 2021, 2:43am UTC](https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518/1 "2021-11-18T02:43:50Z")

</div>

Hello team,  
I have case where information is being displayed in different lines. These lines are not even consecutive lines. Unique field is order ID. I have to triggered email if orderID with Exit came after 30 min of OrderID with Entry.

```auto
2021-11-07 07:11:02.002015|OrderId=100 Entry
2021-11-07 07:11:02.002016|blah1
2021-11-07 07:11:02.002017|blah2
2021-11-07 07:11:02.002018|blah3
2021-11-07 08:11:02.002019|OrderId=100 Exit Symbol=APPLE Price=99 Quantity=100

```

In above log difference between first log (Entry) and last log (exit) is more than 30 min. So we need to trigger mail.

Can you please help me on this.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 18, 2021, 3:19am UTC](https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518/2 "2021-11-18T03:19:09Z")

</div>

Use aggregate filters. A combination of [example 3](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-example3), to get the timeout, and example 2 to disable the timeout if the second line arrives in time.

```
    grok { match => { "message" => "%{TIMESTAMP_ISO8601:[@metadata][ts]}\|OrderId=%{INT:orderId} %{WORD:inOrOut}" } }
    date { match => ["[@metadata][ts]", "YY-MM-dd HH:mm:ss.SSSSSS" ] }

    aggregate {
        task_id => "%{orderId}"
        code => ''
        push_map_as_event_on_timeout => true
        timeout_task_id_field => "orderId"
        timeout => 1800
        timeout_tags => ['_aggregatetimeout']
    }

    if [inOrOut] == "Exit" {
        aggregate {
            task_id => "%{orderId}"
            end_of_task => true
            code => ''
        }
    }

```

Then route the timeout event to an email output...

```
output {
    if "_aggregatetimeout" in [tags] {
        email { ...
```

---

<div class="post-metadata">

**Author:** ![Alex\_Marquardt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_marquardt/32/42925_2.png) [@Alex\_Marquardt](https://discuss.elastic.co/u/Alex_Marquardt)\
**Post date:** [November 18, 2021, 10:43am UTC](https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518/3 "2021-11-18T10:43:31Z")

</div>

You could make use of scripted upserts to do calculate the time difference, as described in: [Using Logstash and Elasticsearch to calculate transaction duration in a microservices architecture](https://alexmarquardt.com/2020/09/16/using-logstash-and-elasticsearch-scripted-upserts-to-calculate-transaction-duration-from-out-of-order-events/)

You could then write an alert to scan for recently completed events where the duration is greater than 30

---

<div class="post-metadata">

**Author:** ![mangeshmj1992](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\
**Post date:** [November 18, 2021, 10:58am UTC](https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518/4 "2021-11-18T10:58:58Z")

</div>

Hello @Badger ,  
I tried following logstash code, but no luck. Duration field not created in entry or exit log

Sample Log line:

```auto
{"timestamp":"2021-11-17T05:52:59:613","level":"info","message":"ENTRY: INTERNAL_HTTP_REQUEST","userId":"rahul","companyId":"abc"}

{"timestamp":"2021-11-17T05:52:59:768","level":"info","message":"EXIT: INTERNAL_HTTP_REQUEST","userId":"rahul","companyId":"abc"}

```

```auto
if [log_processed.message] == "ENTRY: INTERNAL_HTTP_REQUEST" {
        aggregate {
            task_id => "%{log_processed.companyId}"
            code => "map['started'] = event['@timestamp']"
            map_action => "create"
        }
    }

    if [log_processed.message] == "EXIT: INTERNAL_HTTP_REQUEST" {
        aggregate {
            task_id => "%{log_processed.companyId}"
            code => "event['duration'] = event['@timestamp'] - map['started']"
            map_action => "update"
            push_map_as_event_on_timeout => true
            timeout_task_id_field => "log_processed.companyId"
            timeout => 60 # 1 minutes timeout
			timeout_tags => ['_aggregatetimeout']
        }
    }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2021, 10:59am UTC](https://discuss.elastic.co/t/combine-information-from-different-log-lines-into-one-event/289518/5 "2021-12-16T10:59:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.