Python ZOPE Application Framework Integration to Elastic APM

Dixit · October 6, 2022, 1:50pm

Implemented as per your suggestion, please review below and let me know any changes required ---

#@elasticapm.capture_span()
    def do_execute(self, cursor, statement, parameters, context=None):
        #client.begin_transaction('SQL_MXodbc_query')
        with elasticapm.capture_span("sql_statement", span_type='db', span_subtype='sql', span_action='query', extra={
                "db": {"type": "sql", "statement": statement, "instance": getattr(self, "_self_database", None)},
            },skip_frames=1,
            leaf=True):
           cursor.execute(statement, parameters, direct=self._get_direct(context))
        #client.end_transaction("MXODBC_"+statement, "Success")

OUTPUT looks like this ----

Dixit · October 6, 2022, 1:54pm

hi @basepi and @lwintergerst ,

What is the procedure to go for LOG INTEGRATION (Application LOGS) and tie them to the request.

I have application logs (FLASK) and want to trace the API logs with the request?

Thanks,
Dixit

basepi · October 6, 2022, 8:59pm

Glad you got the sql statement working.

You can find documentation about log correlation here: Log correlation | APM Python Agent Reference [6.x] | Elastic

The tl;dr is that you want to use ecs-logging package to format your logs correctly, and then use Elastic Agent or Filebeat to ingest those logs. Then they should show up in the APM logs view.

Dixit · October 7, 2022, 6:21am

hi @basepi ,

Any sample code would help for LOG integration of Python Custom Integration and Flask based Log integration.

Referred, somewhat it's confusing about how - Log correlation | APM Python Agent Reference [6.x] | Elastic

Thanks,
Dixit

lwintergerst · October 10, 2022, 1:00pm

Hi, you can take a look at this example that I shared during the webinar

github.com

LucaWintergerst/observability-example/blob/main/services/06-app-ecs-logging.py

from flask import Flask
import os
from dotenv import load_dotenv

# instrument flask with Elastic APM
from elasticapm.contrib.flask import ElasticAPM
import elasticapm
import random
import time
import structlog

import ecs_logging
import logging
import redis

load_dotenv()

# disable the default flask logger
logger = logging.getLogger('werkzeug')
logger.setLevel(logging.ERROR)

This file has been truncated. show original

I recommend doing it like this, and then as Colton said use Elastic Agent to collect the files

Dixit · October 10, 2022, 2:19pm

hi @lwintergerst and @basepi ,

**Please let me know - log integration for CUSTOM IMPLEMENTATION for the above code/application what I have asked for HELP. **

I have log files in CUSTOM Implementation, and these log files I have to read and correlate.

Can you please help me with a SAMPLE or EXAMPLE like above would help me to grasp .

Flask based am comfortable with now.

Appreciate your help once again here , Thank you @lwintergerst @basepi

Thanks,
Dixit

basepi · October 10, 2022, 4:07pm

@Dixit have you tried applying the examples that we've given to your app? We're happy to help with issues that arise, but unfortunately we don't have the time to integrate it for you.

The way I recommend doing it is implementing structlog in your app (use their linked documentation).

Then it should be as easy as adding two processors, the one from elasticapm, and the one from ecs-logging.

Dixit · October 10, 2022, 4:46pm

Hi @basepi ,

Thanks for your suggestion. I am definitely not expecting for an integration help as it would take your valuable time, I don't want to waste your time.

Need help on sample integration for LOGS - Custom Integrations done in your past so that I can map a reference and try to work on the same lines.

Just a sample copy paste would help me to achieve Elastic APM POC goal and propose for my management for tool adoption .

Hope you can get my point !!

Again thanks for helping me all round !! @basepi @lwintergerst

Thanks,
Dixit

Dixit · October 28, 2022, 2:07pm

hi @lwintergerst and @basepi ,

What is the PRODUCTION OVERHEAD for ELASTIC APM ? Do we have any benchmarks like 5 % or 10% etc.. May be sampling/Instrumentation rate would decide

Thanks,
Dixit

basepi · October 28, 2022, 5:55pm

It's really difficult to measure, but I would say 5% is a pretty reasonable estimate. We work really hard to keep it very lightweight.

Dixit · October 29, 2022, 12:07am

Any examples of measurement would help @basepi

Dixit · November 2, 2022, 4:30pm

hi @basepi and @lwintergerst ,

Is there any way we can truncate the URL in the transactions list so that LONG URLS which has repetitive domain names are removed .
In my custom code instrumentation , the request comes through Apache Web Server and ZOPE captures it , the same request details are sniffed by Python Agent and produces the result/transaction details to APM Server.

For Ex : - My request looks like this -

/VirtualHostBase/https/perf65ci.abc.org:443/cms/ZeUI/VirtualHostRoot/cms/ZeUI/views/Sentinel/SREBase
which is lengthier and have redundant path because of APACHE passing to ZOPE .

Currently the client.end_transaction is like this -

client.end_transaction(request.method + " " + request.get('PATH_INFO'), response.status)

How can I truncate/remove the part which is from APACHE and Domain , only actual path am interested in.

Request your help here please @basepi @lwintergerst

Thank you,

Thanks,
Dixit

basepi · November 2, 2022, 4:54pm

When we're integrating with a framework, typically we like to name the transaction based on the route, not the URL, so that we don't run into cardinality issues like you're seeing. You can probably use a .split() or even a regex to trim your transactions to the relevant information.

Dixit · November 2, 2022, 6:28pm

Any example would help @basepi @lwintergerst

Tried with this and worked out - client.end_transaction(request.method + " " + (request["ACTUAL_URL"]).split("/",3)[3], response.status)

thanks for giving me pointers !! @basepi @lwintergerst

Dixit · November 3, 2022, 3:29pm

Hi @basepi & @lwintergerst ,

How to track the requests which are Served from Kubernetes K8's , is there any agent code we can incorporate where in we get to know how K8's perform & get the metrics as well

Thanks,
Dixit

basepi · November 3, 2022, 5:43pm

We gather a little bit of kubernetes information by default, it should be in the metadata for the transactions. Basically we collect the cgroup metadata for k8s and enrich with environment variables: apm-agent-python/base.py at 98db6462cdb868862f2d96b715fa8b1c01a27274 · elastic/apm-agent-python · GitHub

Dixit · November 7, 2022, 5:42pm

hi @basepi and @lwintergerst ,

I went through ecs_logging & struct_log, somewhat am finding hard to understand.

Need your help please.

In custom instarumentation using "Client" as done above to build traces, how we can attach the exception to the span itself, just like how we can see in flask based elastic APM instrumentation ?

How to list down errors (mostly in logs section , the errors , exception exactly related to the particular transaction)

The code is :

client.begin_transaction('request')
            response = publish(request, module_name, after_list, debug=debug)
            client.end_transaction(request.method + " " + request.get('PATH_INFO'), response.status)

Request life cycle looks like -

Appreciate your help here please @basepi @lwintergerst

basepi · November 8, 2022, 8:09am

Capturing errors is actually very easy! Just use elasticapm.capture_exception() inside of your try:except: block.

Dixit · November 8, 2022, 4:21pm

Hi @basepi ,

Try except should be surrounding client.begin and end calls?

basepi · November 9, 2022, 2:57pm

No, typically you would surround the contents. Something like this:

client.begin_transaction('request')
try:
    response = publish(request, module_name, after_list, debug=debug)
except Exception:
    elasticapm.capture_exception(handled=False)
    raise
finally:
    client.end_transaction(request.method + " " + request.get('PATH_INFO'), response.status)