How to search and display log events linked through a series of UUIDs

Elastic Stack Kibana
1

Hi,

We have an application which communicates with various microservices. Some initial request to an endpoint /api on A might generate several requests to systems B and C, and they themselves might generate further requests D, E and F, each of these requests have a UUID for each side of the transaction.

We end up with logs that are connected through these UUID id fields, where they look like this

{"timestamp":"xx", "request-id":"<UUID>-A",  "response-id":"<UUID>-B", "etc"}
{"timestamp":"xx", "request-id":"<UUID>-A",  "response-id":"<UUID>-C", "etc"}
{"timestamp":"xx", "request-id":"<UUID>-B",  "response-id":"<UUID>-D", "etc"}
{"timestamp":"xx", "request-id":"<UUID>-C",  "response-id":"<UUID>-E", "etc"}

Obviously all these logs form chains of requests, and it would be cool to be able to query on one of those IDs in kibana and get all the associated log events back. However I don't see any obvious way to link them together...

I've been using ingest pipelines to append those ids into a field "ids", which allows me to search on either Id and get transaction connected on both sites, the records end up like:

{"timestamp":"xx", ... , "ids": ["<UUID>-A","<UUID>-B"]}
{"timestamp":"xx", ... , "ids": ["<UUID>-A","<UUID>-C"]}
{"timestamp":"xx", ... , "ids": ["<UUID>-B","<UUID>-D"]}
{"timestamp":"xx", ... , "ids": ["<UUID>-C","<UUID>-E"]}

When running these queries externally, I can just iterate over the results.

So if I search for
ids: "<UUID>-A" will return ["<UUID>-A","<UUID>-B","<UUID>-C"]
I am going to get the first two records back as well. Then I append the additional 2 ids known to those 2 records, and do another search, unitil I have expanded the query and it returns no extra records.
ids: ["<UUID>-A","<UUID>-B","<UUID>-C"] will return all the connected records.
["<UUID>-A","<UUID>-B","<UUID>-C","<UUID>-D"]

in order to know when to finish up, if I recurse the search once, more I get the same number of ids, so I know I have found all the connected records.

I was wondering if there is any way to do this in kibana?

I have started looking at transforms. is this something that could achieve this result ?

Thanks

2

Hi @tolland

This would all be simpler. If you also just had the UUID without the suffix then lots could be done.

The facts that you're putting the application suffix on the end then makes them all unique which makes it much harder.

You could use an ingest pipeline to split the "Combo" UUID into the bade UUID in one field and the app code in another

Or Your apps could the same...

Or you could probably use a runtime field to strip off that end portion into a new field, but at scale that's not super efficient at scale.

In short I would split that somehow

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.