Combine different log events in one data table

Elastic Stack Kibana
1

Hi all,
given are two log files from two different sources.

log 1

2021-05-03 10:00:38,656 INFO  [Service TP] :  | M_ID(3867) | ABC trade received from AFG via MM
2021-05-03 10:00:38,662 INFO  [Service TP] :  | M_ID(3867) | tradeReference: ABC82_00001 | Action Type: N ContractId: ABC82_00001 Version:  | ABC trade mapping done.
2021-05-03 10:00:38,664 DEBUG [Service TP] :  | M_ID(3867) | tradeReference: ABC82_00001 | Action Type: N | for destinations: AAB
2021-05-03 10:00:38,669 DEBUG [Service TP] :  | M_ID(3867) | tradeReference: ABC82_00001 | Action Type: N | DataEnchrichment Status : 15101
2021-05-03 10:00:38,671 INFO  [Service TP] :  | M_ID(3867) | tradeReference: ABC82_00001 | successfully sent to destinations: AAB | MessageID: (2065)

Log 2

2021-05-03 10:00:38,675 INFO  [Service TP] :  | M_ID(2065) | AAB trade received from Host via PK
2021-05-03 10:00:38,684 INFO  [Service TP] :  | M_ID(2065) | tradeReference: ABC82_00001 | Action Type: N ContractId: ABC82_00001 Version:  | AAB trade mapping done.
2021-05-03 10:00:38,687 INFO  [Service TP] :  | M_ID(2065) | tradeReference: ABC82_00001 | Action Type:  | MessageID: (3984)

I have parsed both logs into a single index under the following fields:

* tradeReference : ABC82_00001
* ContractId : ABC82_00001
* log_1_M_ID : 3867
* log_2_M_ID : 2065
* log_1_MessageID : 2065
* log_2_MessageID : 3984
* level : INFO/DEBUG
* Date : the date of every singel line

What I want is the following: a data table where I can track the tradeReference from log_1 with date from log_1 and the same tradeReference from log_2 with date from log_2 and at the end calculate the duration between date from log_1 and date from log_2, here is an example of an expected table:

image

it is an emergency, can someone help with something or give a suggestion.
Thank you very much

2

@bab

This would requires support for a bucket script in Kibana data-tables, which there is not now https://github.com/elastic/kibana/issues/4707

You could run this query manually too in dev-tools.

This would be an example of how to do it:

  • top-level term-aggregation on tradeReference
  • min sub aggregation to get minimum date
  • max sub-aggregation to get maximum date
  • bucket sub-aggregation to subtract max from min
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.