Span-level Custom Context/Metadata

Elastic Observability APM
1

Setup:
Elastic Cloud - Running version 8.6

I am looking for a way to add non-indexed metadata to an individual span instead of just on the transaction itself.
(See Metadata | APM User Guide [8.6] | Elastic)
In certain scenarios I want to be able to log larger json data for triaging issues, however the standard Span.Label is indexed and truncates the text after 1024 characters.
It would be possible to have a singular transaction with the custom content for each action, but that is not ideal since I could be dealing with multiple actions in each transaction and I would like to isolate the custom metadata to the span itself.

2

Bump.

3

Hi @LightSky, welcome to our forum! :wave:
I opened an issue to discuss the options.

1 Like
4

@LightSky Are you able to provide some details on what you are using this for? The use case can help inform whether/what we can do to support this in the APM system. Commenting here or on Eyal's issue is fine. Thanks.

5

Our application is required to log individual actions throughout a given Transaction for each user in our system and the ActionData is a JSON string which can easily exceed 1024 characters, but is usually under 8k or so.
We have other Spans which encompass other logged data aside from the ActionData which we keep inside the Transaction too.
The only workaround we see at the moment is to have a separate Transaction for every single log, which seems like overkill. Or doing a Span.SetLabel with an index (Ex: ActionData[0], ActionData[1]), but that could cause another set of issues since we didn't want to abuse the Elastic Index, and the approach seems kind of hacky.
Finally we saw that Transaction has CustomData support and thought to see if there was a similar approach for Span since we don't need the field indexed whatsoever.

From ElasticAPM's point of view it would look like something similar to this:

  • Transaction
    • Span (1)
      • ActionData: JSON
    • Span (2)
      • Misc Log
    • Span (3)
      • ActionData: JSON
    • ....
    • Span (8)
      • ActionData: JSON
6

@LightSky Thanks. That's very helpful.

How do you use that captured data? Are you using the Kibana APM app to click around and look at the ActionData and other data in the browser UI? Or are you querying the trace indices in Elasticsearch with your own separate tooling?

I'm asking because another possible work-around could be to setup structured log ingest for your applications and use log correlation by ensuring trace.id, span.id, transaction.id fields get added to log records. (I've linked to the Java docs for this. I'm not sure what language(s) are relevant for your usage.) This might not be appealing to you, because it may be more setup/change than you'd like. Also the data linking in the Kibana APM app may not be as convenient.

7

@trentm
We are currently using C#.
We currently use the data via Kibana APM to monitor the data in the browser.
We do however have plans in the future to add support for consuming the data from separate tooling.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.