Getting and Setting Transaction ID

[Alsheh]

We're using the APM Python Flask agent to trace HTTP transactions and collect service metrics and Filebeat+Logstash for collecting service logs. We would like to link logs with the APM metrics:

1. Using the APM transaction ID seems like a good idea but how the transaction ID can be accessed using the APM Flask agent?
2. Moreover, we'd like to replace the APM transaction ID with the Nginx $request_id since Nginx is higher in the stack. This way, Nginx logs, APM metrics, and service logs can all be linked using the Nignx $request_id. How the APM transaction ID can be set using the APM Flask agent such that the set transaction ID gets propagated to the downstream services?
3. Any recommended ways to implement 1 and 2?
4. Any pitfalls that we need to be aware of when setting and getting a transaction ID?

[basepi (Colton Myers)]

Transaction ID in logs is something we definitely want to add/make easier -- in fact, we have https://github.com/elastic/apm-agent-python/issues/520 open already. I'm not certain whether there's a more manual way to do it in the current version -- @beniwohli ?

On (2), I'm not sure if there's a way for us to capture the Nginx request ID, I'll leave that to @beniwohli as well.

[Alsheh]

Thanks for the quick response, @basepi!

For 1 and 2, I was hoping to learn how to access and set the transaction ID in the code, something like this:

...
transaction = execution_context.get_transaction()
# get transation id
old_trans_id =  transaction.get_transaction_id()
# set transaction id
new_trans_id = '1234567890'
transaction.set_transaction_id(new_trans_id)
...

Currently, we're passing the Nginx request ID in a custom header, and, supporting that out of the box would be nice, but, for the time being, we would be interested to manually set the Nginx request id as an APM transaction id using the transaction id setter.

[basepi (Colton Myers)]

Ah! I think that's actually pretty easy. I think when you are talking about "transaction ID", I think what you actually want is "transaction name". Turns out there's a nice API call for just this purpose: https://www.elastic.co/guide/en/apm/agent/python/current/api.html#api-set-transaction-name

I think that will get you where you want to go.

[Alsheh]

Actually, I would like to keep the transaction name as is which usually corresponds to <HTTP-method> <API-Endpoint> for my micro-service; I have looked at the docs you provided for an API call that can do elasticapm.get_transaction_id() and elasticapm.set_transaction_id() but I couldn't find any. I'm wondering if there is any undocumented API call that can be used to set and get the transaction id?

[beniwohli (Benjamin Wohlwend @ APM)]

Hi @Alsheh

custom transaction IDs is not something we plan to support I'm afraid. The transaction and trace IDs follow the format defined in the W3C Trace Context standard. Providing APIs to change the ID would risk breaking our conformity with that standard.

Instead, I suggest to tag the transaction with the nginx request ID, something like (assuming you configured nginx to add an "nginx-request-id" header to the request)

import elasticapm

elasticapm.tag(nginx_request_id=request.headers["nginx-request-id"])

This code could e.g. be hooked up to the request_started signal. Just make sure to only register the signal after the ElasticAPM call, this ensures that our signal handler (which creates the transaction object) is called before your handler (which then tags said transaction object).

[Alsheh]

@beniwohli this is useful, thanks! I see how setting a custom trace ID could result in unexpected results but would it be possible to get the generated APM trace ID in the code for logging purposes?

[basepi (Colton Myers)]

That's where that issue link I noted above comes in. We're going to provide a logging filter that will add the transaction ID and other useful information into your log records, so you can format them into your logs. Keep an eye on that issue for more details.

[Alsheh]

@basepi thanks for the clarification! Until a PR is merged for the logging filter, isn't there a way to get the trace id from a transaction so I can manually add it to the log record?

[basepi (Colton Myers)]

Yep. It's a touch hairy, since it's not designed as a public API, but here's an example (untested, I'm in the process of implementing that issue we've been discussing):

from elasticapm.traces import execution_context

transaction = execution_context.get_transaction()
id = transaction.id
trace = transaction.trace_parent.trace_id if transaction.trace_parent else None

[basepi (Colton Myers)]

I just wanted to update that we just merged some public API helpers for these IDs, as well as some logging filters and a structlog processor that can make adding these IDs to your logs super easy. https://github.com/elastic/apm-agent-python/pull/586

[Alsheh]

Hi @basepi! thanks for the update! I found the update on the documentation. It looks like it hasn't been released, yet, any ideas when to expect the release that will have these logging updates?

[basepi (Colton Myers)]

Yep! We'll be releasing alongside the Elastic Stack 7.4.0 release, which we are expecting soon.