# Combine different log events in one data table

**URL:** <https://discuss.elastic.co/t/combine-different-log-events-in-one-data-table/272403>\
**Category:** Kibana\
**Created:** [May 7, 2021, 10:14am UTC](https://discuss.elastic.co/t/combine-different-log-events-in-one-data-table/272403 "2021-05-07T10:14:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bab/32/86201_2.png) [@bab](https://discuss.elastic.co/u/bab)\
**Post date:** [May 7, 2021, 10:14am UTC](https://discuss.elastic.co/t/combine-different-log-events-in-one-data-table/272403/1 "2021-05-07T10:14:40Z")

</div>

Hi all,  
given are two log files from two different sources.

log 1

> ```
> 2021-05-03 10:00:38,656 INFO [Service TP] : | M_ID(3867) | ABC trade received from AFG via MM
> 2021-05-03 10:00:38,662 INFO [Service TP] : | M_ID(3867) | tradeReference: ABC82_00001 | Action Type: N ContractId: ABC82_00001 Version: | ABC trade mapping done.
> 2021-05-03 10:00:38,664 DEBUG [Service TP] : | M_ID(3867) | tradeReference: ABC82_00001 | Action Type: N | for destinations: AAB
> 2021-05-03 10:00:38,669 DEBUG [Service TP] : | M_ID(3867) | tradeReference: ABC82_00001 | Action Type: N | DataEnchrichment Status : 15101
> 2021-05-03 10:00:38,671 INFO [Service TP] : | M_ID(3867) | tradeReference: ABC82_00001 | successfully sent to destinations: AAB | MessageID: (2065)
> 
> ```

Log 2

> ```
> 2021-05-03 10:00:38,675 INFO [Service TP] : | M_ID(2065) | AAB trade received from Host via PK
> 2021-05-03 10:00:38,684 INFO [Service TP] : | M_ID(2065) | tradeReference: ABC82_00001 | Action Type: N ContractId: ABC82_00001 Version: | AAB trade mapping done.
> 2021-05-03 10:00:38,687 INFO [Service TP] : | M_ID(2065) | tradeReference: ABC82_00001 | Action Type: | MessageID: (3984)
> 
> ```

I have parsed both logs into a single index under the following fields:

```
* tradeReference : ABC82_00001
* ContractId : ABC82_00001
* log_1_M_ID : 3867
* log_2_M_ID : 2065
* log_1_MessageID : 2065
* log_2_MessageID : 3984
* level : INFO/DEBUG
* Date : the date of every singel line

```

What I want is the following: a data table where I can track the tradeReference from log\_1 with date from log\_1 and the same tradeReference from log\_2 with date from log\_2 and at the end calculate the duration between date from log\_1 and date from log\_2, here is an example of an expected table:

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/5/45ff4ce1d78620f2e31358242494129682affef3.png)

it is an emergency, can someone help with something or give a suggestion.  
Thank you very much

---

<div class="post-metadata">

**Author:** ![thomasneirynck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomasneirynck/32/23313_2.png) [@thomasneirynck](https://discuss.elastic.co/u/thomasneirynck)\
**Post date:** [May 11, 2021, 2:04pm UTC](https://discuss.elastic.co/t/combine-different-log-events-in-one-data-table/272403/2 "2021-05-11T14:04:40Z")

</div>

@bab

This would requires support for a bucket script in Kibana data-tables, which there is not now [https://github.com/elastic/kibana/issues/4707](https://github.com/elastic/kibana/issues/4707)

You could run this query manually too in dev-tools.

This would be an example of how to do it:

> [@Elasticsearch bucket\_script to subtract two timestamps](https://discuss.elastic.co/t/elasticsearch-bucket-script-to-subtract-two-timestamps/125309/2):
>
> Do you need those firstUrl and lastUrl aggregations to be terms aggregations? Because if you change those to min and max aggregations the solution is simple: { "query": { "bool": { "filter": { "range": { "page.time": { "gte": "now-w", "lte": "now" } } } } }, "aggs": { "sessionLengthData": { "terms": { "field": "sId.keyword", "size": 10 }, "aggs": { "firstUrl": { …

- top-level term-aggregation on `tradeReference`
- min sub aggregation to get minimum date
- max sub-aggregation to get maximum date
- bucket sub-aggregation to subtract max from min

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2021, 2:05pm UTC](https://discuss.elastic.co/t/combine-different-log-events-in-one-data-table/272403/3 "2021-06-08T14:05:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.