How to Group Spans, Metadata and Transactions Together for Visualization

Elastic Observability APM
1

Kibana version: 7.3

Elasticsearch version: 7.3

APM Server version: 7.3

APM Agent language and version: Intake API (Building new agent)

Hello,

I am looking for a way to link spans, transactions and metadata all together in Kibana so I can properly use information from the metadata to group the spans.

I'll give a specific example of what I'm trying to accomplish. I'm trying to show the average duration of a specific span on a line graph, where each line is a different version number. To do this I first filter for the span.name that I am wanting to track:
image

Then I add average span duration to the y axis:
image

Next I make the x axis a date histogram so that I can see this data over a period of time:
image

Now here comes the problem. I need to split the lines by version number so I can see how long specific version numbers were taking at a given time but since the data is tracking the spans and not the metadata, the version tag in the metadata is not accessible to be sorted by.

Even though the spans were all sent in with the metadata tag, there is no value that links these two events and even if there was I wouldn't know how to link them visually in Kibana.

My hacky solution was to pass the version number as a label in the spans context for every single span. Then the data exists as a label for the spans so I can track it:

image
image.png934×419 19.2 KB

This method however is not scalable. Lets say in the future I want to group spans by their parent transactions, transaction type. Even though there is a parent id relationship between a span and it's transaction, I have no way to visually link the two together in Kibana.

Does anyone know of a way to handle this issue?

Thank you,

Matt

2

Welcome to the community @LordMathis. For the specific example in this question, you might consider creating a separate index of deployments and showing an annotation per version. This won't be perfect if deployments overlap but should provide some insight over time/version changes.

As for the general question, I think you'll have to do some external processing to make that happen, such as your version labels. Problems are typically tackled from the transaction level though I understand your use case here and will give it more thought and hopefully others will chime in as well.

What kind of agent are you developing?

3

We program on a language called Delphi, which is the object oriented version of Pascal. We have a relatively holistic implementation of the agent we are just wrapping up a few issues. Mainly the one posted above and the fact that metric gathering in Delphi is horrendous.

Thank you for the reply @gil!

4

Okay this is a good answer. I think this will just require some more thought and consideration on our end as to how we can properly track these things together.

If anyone has any ideas or suggestions as to how to handle these cross event sorting issues please feel free to add on.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.