How to add contextual information with Flask plugin?

[fr0der1c]

I started to use Flask APM package today. It worked fine. But when I tried to add contextual information, I got into trouble.

I looked for official doc (Flask support | APM Python Agent Reference [2.x] | Elastic) and got nothing. Then I started looking into site-packages/elasticapm/contrib/flask/__init__.py and found that there isn't any method in ElasticAPM class to add contextual information. (The elasticapm package does have a series of methods like set_user_context though)

Since there is already a plugin for Flask, I don't want to make a wheel again. So I started hacking the package. Here is how I think and what I've done:

The request_finished method in ElasticAPM class connected to Flask's request_finished signal. And when a request is finished, the signal would be triggered. So what I need to do is to add contextual information before request_finished method get executed, by using a monkey patch to this method.

Here the original request_finished method:

    def request_finished(self, app, response):
        if not self.app.debug or self.client.config.debug:
            ...[omitted]...
            elasticapm.set_transaction_name(rule, override=False)
            elasticapm.set_transaction_result(result, override=False)
            # Instead of calling end_transaction here, we defer the call until the response is closed.
            # This ensures that we capture things that happen until the WSGI server closes the response.
            response.call_on_close(self.client.end_transaction)

Here is my app/__init__.py:

from .utils import monkey_patch
apm = ElasticAPM()
apm.request_finished = monkey_patch.elastic_apm_request_finished(apm.request_finished)


def create_app():
    app = Flask(__name__)
    app.config.from_object(config)
    apm.init_app(app,
                 service_name=app.config['ELASTIC_APM']['SERVICE_NAME'],
                 secret_token=app.config['ELASTIC_APM']['SECRET_TOKEN'],
                 server_url=app.config['ELASTIC_APM']['SERVER_URL'])
    ...
    return app

And my monkey_patch.py:

import functools
import elasticapm
from flask import request


def elastic_apm_request_finished(original_func):
    @functools.wraps(original_func)
    def _patched(self, app, response):
        if not self.app.debug or self.client.config.debug:
            # add user_id in context
            elasticapm.set_user_context(user_id=request.cookies.get('UM_distinctid', None))

        # execute the original `request_finished` function
        original_func(self, app, response)

    return _patched

It should work as expected. But unfortunately it didn't. I got

TypeError: _patched() missing 1 required positional argument: 'app'

After a quite-deep look into the code, I found the reason for the error.

The blinker (which is the package handles the signals) only pass two parameters when invoking the request_finished. app as positional parameter, and response as named parameter. As a method in a class, Python should add parameter self automatically. But it is not actually adding this parameter, after I decorated the request_finished method. So the actual app parameter is taking the position of self, and the second second parameter gets nothing!

I'm working with this problem for hours and getting quite frustrated. Does anyone knows where I'm wrong, or is there any better way to add contextual information?

[fr0der1c]

Okey...I finally got the solution.

The monkey patch should be applied on the method of the class, not the method of the instance of the class. Methods of instance of the class is type BoundMethod, which has __self__ and __func__ attribute. When you do monkey patch on it, it's replaced with a normal function (you can use type(instance) to see the difference).

In short, monkey patch by:

ElasticAPM.request_finished = monkey_patch.elastic_apm_request_finished(ElasticAPM.request_finished)

not

apm.request_finished = monkey_patch.elastic_apm_request_finished(apm.request_finished)

will solve the problem.

Hope that APM team improve Flask plugin so we don't need to use such this geek way to add contextual information in the future.

[simitt (Silvia Mitter)]

Thanks for trying out Elastic APM!
You might want to take a look at the Public API section on how to set custom context and add tags. I hope that helps.

[fr0der1c]

Thanks for the link.

I actually looked at these docs before hacking into the package. However users are unable to use Public API if they're using the Flask APM plugin. I think the Flask plugin needs to provide a hook to allow user to use Elastic APM Public API.

[simitt (Silvia Mitter)]

Here is a little snippet combining Elastic APM in a flask app and adding some custom context:

import elasticapm
from flask import Flask
from elasticapm.contrib.flask import ElasticAPM

app = Flask(__name__)

app.config['ELASTIC_APM'] = {
    'DEBUG': True,
    'SERVER_URL': 'http://localhost:8200',
    'SERVICE_NAME': 'test-service',
}
apm = ElasticAPM(app)


@app.route('/')
def index():
    elasticapm.set_custom_context({'my-context': 123})
    return 'OK'

Please let me know what you are missing from this.

[fr0der1c]

Setting context in Flask view functions in your way works for one route. However if I want to set a tag or context for all requests (In my case, I read userid from cookies and add to user context), I have no option but to do some hacking.

By the way, what's the difference between tags and custom context? They look pretty alike.

[beniwohli (Benjamin Wohlwend @ APM)]

@fr0der1c a somewhat less hacky solution could be to use a middleware

class UserContextMiddleware(object):
    def __init__(self, app):
        self.app = app

    def __call__(self, environ, start_response):
        resp = self.app(environ, start_response)
        elasticapm.set_custom_context({'user-id': 1234})
        return resp


app.wsgi_app = TaggingMiddleware(app.wsgi_app)

One drawback with this solution is that request isn't available in the middleware, so you'd need to grab the cookie from environ.

As for the difference between custom context and tags: the custom context can be any JSON serializable object (e.g. nested data). However, it is not indexed in Elasticsearch, so you can't search/filter/aggregate on it. Tags are flat key/value string pairs, and they are indexed in Elasticsearch.