Displaying data from one event to other events

Elastic Stack Kibana
1

Hello All

Is there a way in kibana to display fields from one event to other related events in data tables aggregations and discover views?

for example, 2 events with this format:

1. {
  id: 1,
  city: 'London',
  temperature : '20',
  //.....other fields.../
}

2. {
  id: 1,
  city: 'London',
  //.....other fields... (except temperature) //
}

and I want to display the temperature for the second event given that the events are linked through the id field

Is there a way to do that in kibana?

I searched and found the only way is by de-normalizing the data and propagate
the temperature to the other events but the problem is we don't want to control the data from the source system

Also I tried using logstash for that but the events are not guaranteed to come in specific order and the rate of the events are high so accessing Elasticsearch to get the required data and populate it is also not guaranteed because the event may still not been indexed

I thought the only way is to use another UI framework and control what data to retrieve and what to display (by using a backend layer on top of it to do that)

Any suggestions?

Thank You

2

You can ask logstash Elasticsearch output plugin to join the 2 events into the same event with _id = 1 and fields city & temperature

3

Something like this

output {
	elasticsearch {
                hosts       		=> "***"
                user        		=> "***"
                password    		=> "***"
				index       		=> "***"
				document_id 		=> "%{[id]}"
				action 				=> "update"
				doc_as_upsert      	=> true
		}
}
4

But I want to keep all the events separate to be able to display the other fields in separate rows (those other fields may differ from one event to another for the same id but the temperature and city will always be the same for the same id)

5

It's not clear your use case, can you provide a concrete example ?

6

I was trying to simplify the use case with an example but the example may seem unrealistic so I will explain again with the actual case we have

we have two events below

1. {
  shipmentId: 1,
  trailerId: 10,
  eventName: ShipmentLoaded,
  eventTimestamp:  2021-10-01T10:00:00
}

2. {
  shipmentId: 1,
  eventName: ShipmentPrepared,
  eventTimestamp: 2021-10-01T10:00:05
}

we know for a fact that any event with name "ShipmentLoaded" will always have "trailerId" field

And one trailer may contain multiple shipments

so the second event (which is having the same shipmentId as the first event) is also belonging to trailerId 10. But this second event is coming from the source system without this trailerId field in it

so I want to show a data table like this:

id        trailer id        eventName                             eventTimestamp
1          10                  ShipmentLoaded                   2021-10-01T10:00:00
1          10                  ShipmentPrepared                2021-10-01T10:00:05
7

Does this make sens ?

id        trailer id        ShipmentLoaded_time                             ShipmentPrepared_time
1          10               2021-10-01T10:00:00								2021-10-01T10:00:05
8

I'm thinking about tranform that will allow you to have an entity centric index around shipments

9

Thanks I will look into that

So for that use case the only possible tables we can get are summary tables, not detailed ones?

10

Since the lifecyle of a shipment is limited i guess, a shipment can goes through a limited number of steps, a entity centric index may be the good approach for that use case
You can explore other scenarios of enrichiment using Elasticsearch filter in logstash but may not be scalable

11

Thank you. I will explore the entity centric option

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.