Python FLASK Elastic APM Agent issues - Culprit threading._bootstrap

Elastic Observability APM
1

Hi @basepi @lwintergerst ,

We are using ELASTIC APM to instrument FLASK APP with Wsgi, Gunicorn.

The events are generated and submitted to APM Server with status code as "202". The APM - Kibana UI is unable to display the requests.

Error Message :-
Log message

Sent request, url=http://X.X.X.7.149:8200/intake/v2/events size=5.11kb status=202
Exception message : N/A
Culprit - threading._bootstrap

image
image817×341 17.6 KB

image
image993×135 23.8 KB

image
image1312×275 29.5 KB

@basepi can you please help me to resolve this issue ?

Thanks

2

Hey @Dixit

I don't think these are actual errors, but log messages that are picked up by our log shipper. Do you per chance initialize the ElasticAPM app with logging=True? We recommend to not use True, but specify a log level, e.g. logging.ERROR.

For shipping general logs to Elastic, we recommend to use e.g. filebeat.

3

hi @beniwohli ,

Thanks for your reply .

Yes, I did enabled "logging=True" and initialized agent. Will try with "logging.ERROR".

My question is : The APM UI shows empty with no transactions being recorded, that's the issue am facing even though APM logs show - {"log.level":"info","@timestamp":"2022-11-15T04:47:07.194-0600","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-python/6.11.0 (RHEL_53_WorklistServices)","source.address":"172.27.7.53","http.request.body.bytes":526,"http.request.id":"10f63edd-8dbe-400b-af74-b46c86f62c5a","event.duration":10410600,"http.response.status_code":202,"ecs.version":"1.6.0"}

APM UI shows nothing -

image
image1035×297 9.47 KB

can you please help me why this happens ? @beniwohli @basepi

4

hi @beniwohli @basepi

Falsk app with Gunicorn + wsgi uses gevent - gunicorn --worker-class=gevent --timeout=0 --worker-connections=100 --workers=9 wsgi:app

5

Can you show us your initialization of your flask app? I assume you followed the docs here?

6

hi @basepi

We are initializing the app like here -

*[abd@KAhhhhhFKA01 worklist]$ cat wsgi.py*
*from worklist.app import create_app*

*from flask import Flask*
*from elasticapm.contrib.flask import ElasticAPM*
*import elasticapm*
*from elasticapm.handlers.logging import LoggingHandler*
*from elasticapm.contrib.flask import ElasticAPM*
*from elasticapm.handlers.logging import LoggingHandler*
*"""*
*# WSGI - stands for web server gateway interface.*
*# used to forword requests from web server to a backend python web*
*  application or framework.*
*# WSGI server simply invokes a callable object on the WSGI application*
*"""*
*server_url = 'http://x.x.x.x:8200'*
*service_name = 'RHEL_WorklistSVC'*
*environment = 'RHEL_DEV'*
*app = create_app()*
*apm = ElasticAPM(app, server_url=server_url, service_name=service_name, environment=environment, logging=True)*
*elasticapm.instrument()*

Not sure what changes to be done.

Please help @basepi @beniwohli @lwintergerst

7

Can you share the create_app() code? I don't see any red flags here. Very surprised you're getting log messages but not traces.

8

hi @basepi ,

Here is the code for "CREATE_APP" , please help me why we are not getting transactions in the UI.

[jaaaa@PXXXXXXXKA01 worklist]$ cat app.py
import os
from datetime import timedelta
from flask import request
from werkzeug.exceptions import HTTPException
from flask_jwt_extended import JWTManager
from flask_openapi3 import Info, HTTPBearer, OpenAPI
from jivacore.logger.log import Logger
from worklist.resources.worklist import worklist_routes
from worklist.resources.other_routes import health_routes
from worklist.common import constants
from worklist.security import permissions
from jivacore.db.db import get_scoped_session
from worklist import config, error_handlers


def create_app(flask_config=None, **kwargs):
    """
    Instantiation and configure of flask application
    Args:
        flask_config (str): Flask configuration value
    Returns:
        OpenAPI: Flask OpenAPI instance
    """
    info = Info(title=os.getenv("PROJECT_NAME"), version="1.0.0", description=constants.WORKLIST_DESCRIPTION)
    app = OpenAPI(__name__, info=info, security_schemes={"jwt": HTTPBearer()})

    # Configuring logger for Worklist Service
    logger = Logger(request)
    logger.info('*******************Starting worklist Service********************')

    # Load application configuration from the object
    config.LoadApplicationConfig(app, flask_config)

    # Adding permissions to the application
    app.permissions = permissions

    # Adding scoped_session to the application
    app.scoped_session = get_scoped_session()

    # JSON Web Token configuration
    jwt = JWTManager(app)
    app.config["JWT_IDENTITY_CLAIM"] = "jti"
    app.config["JWT_TOKEN_LOCATION"] = ["headers", "cookies"]
    app.config["JWT_ACCESS_COOKIE_NAME"] = "aptn"
    app.config["JWT_ACCESS_CSRF_HEADER_NAME"] = "X-CSRFT"
    app.config["JWT_ACCESS_TOKEN_EXPIRES"] = timedelta(minutes=30)

    # Register Error Handlers
    jwt.unauthorized_loader(error_handlers.missing_token_callback)
    app.register_error_handler(HTTPException, error_handlers.http_errors)
    app.register_error_handler(Exception, error_handlers.global_errors)

    # Register blueprints
    app.register_api(worklist_routes)
    app.register_api(health_routes)

    # Before and after request handlers
    app.before_request(config.before_request_func)
    app.after_request(config.after_request_func)
    app.teardown_request(config.teardown_request_func)

    return app

Have used all agents version starting from 6.9 to latest, no luck.

@basepi @beniwohli @lwintergerst

9

hi Colton @basepi,

Please find the create app. Please let us know what is causing the UI to show blank.

Appreciate your help here.

[jaaaa@PXXXXXXXKA01 worklist]$ cat app.py
import os
from datetime import timedelta
from flask import request
from werkzeug.exceptions import HTTPException
from flask_jwt_extended import JWTManager
from flask_openapi3 import Info, HTTPBearer, OpenAPI
from jivacore.logger.log import Logger
from worklist.resources.worklist import worklist_routes
from worklist.resources.other_routes import health_routes
from worklist.common import constants
from worklist.security import permissions
from jivacore.db.db import get_scoped_session
from worklist import config, error_handlers

def create_app(flask_config=None, **kwargs):
"""
Instantiation and configure of flask application
Args:
flask_config (str): Flask configuration value
Returns:
OpenAPI: Flask OpenAPI instance
"""
info = Info(title=os.getenv("PROJECT_NAME"), version="1.0.0", description=constants.WORKLIST_DESCRIPTION)
app = OpenAPI(name, info=info, security_schemes={"jwt": HTTPBearer()})

Configuring logger for Worklist Service

logger = Logger(request)
logger.info('Starting worklist Service*')

Load application configuration from the object

config.LoadApplicationConfig(app, flask_config)

Adding permissions to the application

app.permissions = permissions

Adding scoped_session to the application

app.scoped_session = get_scoped_session()

JSON Web Token configuration

jwt = JWTManager(app)
app.config["JWT_IDENTITY_CLAIM"] = "jti"
app.config["JWT_TOKEN_LOCATION"] = ["headers", "cookies"]
app.config["JWT_ACCESS_COOKIE_NAME"] = "aptn"
app.config["JWT_ACCESS_CSRF_HEADER_NAME"] = "X-CSRFT"
app.config["JWT_ACCESS_TOKEN_EXPIRES"] = timedelta(minutes=30)

Register Error Handlers

jwt.unauthorized_loader(error_handlers.missing_token_callback)
app.register_error_handler(HTTPException, error_handlers.http_errors)
app.register_error_handler(Exception, error_handlers.global_errors)

Register blueprints

app.register_api(worklist_routes)
app.register_api(health_routes)

Before and after request handlers

app.before_request(config.before_request_func)
app.after_request(config.after_request_func)
app.teardown_request(config.teardown_request_func)

return app

Thank you

10

My bet is that it's because you're using flask_openapi3 rather than flask. In our code, we use signals to create transactions.

However, from your code it may be that those handlers need to be registered in a different way for flask_openapi3.

Can you please open an issue for flask_openapi3 support?

11

hi @basepi @beniwohli @lwintergerst ,

Please help me with this issue at least a workaround for FLASKAPI3 support in Elastic APM.

ISSUE RAISED for Elastic APM - FLASK - Handlers need to be registered in a different way for flask_openapi3. #1697

GITHUB ISSUE LINK - apm-agent-python/issues/1697

https://github.com/elastic/apm-agent-python/issues/1697

12

hi @basepi , @beniwohli ,

is there any temporary workaround for above issue , so that I can move forward till I get a fix for flask_openapi3 ?

Appreciate your help here please @beniwohli @basepi @lwintergerst

Thank you

13

I'm not sure, I'll need to dive into the flask_openapi3 code and we probably won't have time to do that until our next release cycle.

In the meantime you could use our existing flask support as a reference and see if you can get it working using openapi3 signals.

Apologies for the inconvenience!

14

Hi @basepi

Is there anything else you suggest to make it work.

When tentatively the release would happen for python flask agent.

@basepi, request to help me with some pointers to proceed further.

Thanks

15

Hi @basepi,

Is there any timelines where “my issue” GitHub would be considered and fixed?

What would’ve the typical timelines

16

Somehow @basepi if you make it work or some tweaks/workaround would appreciate.

Pretty much stuck here though!!

17

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.