Use Kibana to find API endpoints with most number of inter-service calls

[feicipet (Wong CS)]

Hi,

I have a bunch of microservices that are calling each other. Say I have serviceA, serviceB and serviceC, the call sequence may look like:

API GW -> ServiceA.endpointA1 -> ServiceB.endpointB1 -> ServiceC.endpointC2

Each endpoint invoked will send a log to ES. In the logs that ship to ES, I have the following fields:

1. api.correlationid - this is a common correlation ID that links all endpoints invoked in one call flow (similar to Zipkin's traceID)
2. api.command - e.g. POST /v1/login
3. api.source - This indicates the source where the call was made from. If ServiceB gets a call from ServiceA, in ServiceB request logs, api.source would denote "ServiceA" (kinda like a referrer)

With this information, I have traceability throughout all the microservices for each individual request.

But now I want to aggregate the information to show, for e.g.: Which first contact endpoint (the endpoint that was invoked directly from API Gateway) has the most number of inter-service calls following it.

Aggregating the count of API calls by api.correlationid is easy enough to do but having a bunch of arbitrary correlation IDs is not very useful. I want to further aggregate these correlation IDs under the api.command they are associated to (kind of like a subquery).

Is this possible in Kibana?

Thanks
Wong

[spalger (Spencer Alger)]

Do you have a reliable way to sort the events so that the first event would be the first contact endpoint? If so you might be able to use a "top hits" metric and concatenate the api.command from the first event within each api.correlationid bucket.