Merging Kibana Results

Elastic Stack Kibana
1

I have two logs from two different applications writing to the same index.
Log1:

[
"Processed Payload - UID: c9c9502f-f398-434f-3d2efc7853a3, AlertId : 000130114"]

Log2:

["Processed Payload - UID: c9c9502f-f398-434f-3d2efc7853a3, ClientId: 283400, TrackId: tracking-456, File: transaction.csv"]

From the UID in log2 i need to check if there are any matching alertId in log1 and if found i need to merge the data and get it in the below format.

Processed Payload - UID: c9c9502f-f398-434f-3d2efc7853a3, ClientId: 283400, TrackId: tracking-456, File: transaction.csv", AlertId:000130114

How can i do this ??
I will need this to be running as part of Kibana watcher

2

As you may know, Elasticsearch is not meant to run JOIN queries at runtime.

I think I would approach this as:

  • Process your incoming data to extract the identifiers using an ingest pipeline, maybe even using the reroute processor to split your logs in different indices.
  • Create a pivot transform that pivots (merges) the entries with a common UID. You can probably even set up another ingest pipeline in the new output index to generate that format message but you probably want to do that later.

I'm unsure how this plays with your Kibana Watcher use case, but it seems worth exploring.