hi @basepi & @lwintergerst ,
We are experiencing lot of exceptions/errors in the Zope application and even though adding "Above Code" doesn't display exceptions/errors in the APM UI.
If logging and Errors are taken care then pretty much am done with Elastic APM implementation for Zope Python framework.
Last thing need a help though @basepi @lwintergerst !! Appreciate it.
Looks like you're seeing the transactions, which leads me to believe your Client object is configured correctly. Could you show me your code? I want to make sure we didn't miss something between my example and your implementation.
hi @basepi ,
Here is the code in full , where we use Zope Framework. Help me to understand what am missing here @basepi @lwintergerst @beniwohli
####### ELASTIC APM INTEGRATION #######
import elasticapm
logger = get_logger("elasticapm.errors.client")
client = elasticapm.Client(service_name="JAAAVA-CI-NSTANCE", framework_name="Zope", framework_version="4.0", server_url="http://APM93.zzzzga:8200")
elasticapm.instrument()
_default_debug_mode = False
_default_realm = None
def call_object(object, args, request):
return object(*args)
def missing_name(name, request):
if name == 'self':
return request['PARENTS'][0]
request.response.badRequestError(name)
def dont_publish_class(klass, request):
request.response.forbiddenError("class %s" % klass.__name__)
def validate_user(request, user):
newSecurityManager(request, user)
def set_default_debug_mode(debug_mode):
global _default_debug_mode
_default_debug_mode = debug_mode
def set_default_authentication_realm(realm):
global _default_realm
_default_realm = realm
def publish(request, module_name, after_list, debug=0,
# Optimize:
call_object=call_object,
missing_name=missing_name,
dont_publish_class=dont_publish_class,
mapply=mapply,
):
(bobo_before, bobo_after, object, realm, debug_mode, err_hook,
validated_hook, transactions_manager) = get_module_info(module_name)
parents = None
response = None
try:
notify(pubevents.PubStart(request))
# TODO pass request here once BaseRequest implements IParticipation
newInteraction()
request.processInputs()
request_get = request.get
response = request.response
# First check for "cancel" redirect:
if request_get('SUBMIT', '').strip().lower() == 'cancel':
cancel = request_get('CANCEL_ACTION', '')
if cancel:
# Relative URLs aren't part of the spec, but are accepted by
# some browsers.
for part, base in zip(urlparse(cancel)[:3],
urlparse(request['BASE1'])[:3]):
if not part:
continue
if not part.startswith(base):
cancel = ''
break
if cancel:
raise Redirect(cancel)
after_list[0] = bobo_after
if debug_mode:
response.debug_mode = debug_mode
if realm and not request.get('REMOTE_USER', None):
response.realm = realm
noSecurityManager()
if bobo_before is not None:
bobo_before()
# Get the path list.
# According to RFC1738 a trailing space in the path is valid.
path = request_get('PATH_INFO')
request['PARENTS'] = parents = [object]
if transactions_manager:
transactions_manager.begin()
object = request.traverse(path, validated_hook=validated_hook)
if IBrowserPage.providedBy(object):
request.postProcessInputs()
notify(pubevents.PubAfterTraversal(request))
if transactions_manager:
recordMetaData(object, request)
result = mapply(object, request.args, request,
call_object, 1,
missing_name,
dont_publish_class,
request, bind=1)
if result is not response:
response.setBody(result)
notify(pubevents.PubBeforeCommit(request))
if transactions_manager:
transactions_manager.commit()
notify(pubevents.PubSuccess(request))
endInteraction()
return response
except:
# save in order to give 'PubFailure' the original exception info
exc_info = sys.exc_info()
# DM: provide nicer error message for FTP
sm = None
if response is not None:
sm = getattr(response, "setMessage", None)
if sm is not None:
from asyncore import compact_traceback
cl, val = sys.exc_info()[:2]
sm('%s: %s %s' % (
getattr(cl, '__name__', cl), val,
debug_mode and compact_traceback()[-1] or ''))
# debug is just used by tests (has nothing to do with debug_mode!)
if not debug and err_hook is not None:
retry = False
if parents:
parents = parents[0]
try:
try:
return err_hook(parents, request,
sys.exc_info()[0],
sys.exc_info()[1],
sys.exc_info()[2],
)
except Retry:
if not request.supports_retry():
return err_hook(parents, request,
sys.exc_info()[0],
sys.exc_info()[1],
sys.exc_info()[2],
)
retry = True
finally:
# Note: 'abort's can fail.
# Nevertheless, we want end request handling.
try:
try:
notify(pubevents.PubBeforeAbort(
request, exc_info, retry))
finally:
if transactions_manager:
transactions_manager.abort()
finally:
endInteraction()
notify(pubevents.PubFailure(request, exc_info, retry))
# Only reachable if Retry is raised and request supports retry.
newrequest = request.retry()
request.close() # Free resources held by the request.
# Set the default layer/skin on the newly generated request
if ISkinnable.providedBy(newrequest):
setDefaultSkin(newrequest)
try:
return publish(newrequest, module_name, after_list, debug)
finally:
newrequest.close()
else:
# Note: 'abort's can fail.
# Nevertheless, we want end request handling.
try:
try:
notify(pubevents.PubBeforeAbort(request, exc_info, False))
finally:
if transactions_manager:
transactions_manager.abort()
finally:
endInteraction()
notify(pubevents.PubFailure(request, exc_info, False))
raise
@elasticapm.capture_span()
def publish_module_standard(
module_name,
stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr,
environ=os.environ, debug=0, request=None, response=None):
must_die = 0
status = 200
after_list = [None]
try:
try:
if response is None:
response = Response(stdout=stdout, stderr=stderr)
else:
stdout = response.stdout
# debug is just used by tests (has nothing to do with debug_mode!)
response.handle_errors = not debug
if request is None:
request = Request(stdin, environ, response)
setRequest(request)
# make sure that the request we hand over has the
# default layer/skin set on it; subsequent code that
# wants to look up views will likely depend on it
if ISkinnable.providedBy(request):
setDefaultSkin(request)
########################### Elastic APM ####################
client.begin_transaction('request')
try:
response = publish(request, module_name, after_list, debug=debug)
except Exception:
client.capture_exception()
raise
finally:
client.end_transaction(request.method + " " + request["ACTUAL_URL"], response.status)
########################### Elastic APM ####################
except (SystemExit, ImportError):
# XXX: Rendered ImportErrors were never caught here because they
# were re-raised as string exceptions. Maybe we should handle
# ImportErrors like all other exceptions. Currently they are not
# re-raised at all, so they don't show up here.
must_die = sys.exc_info()
request.response.exception(1)
except:
# debug is just used by tests (has nothing to do with debug_mode!)
if debug:
raise
request.response.exception()
status = response.getStatus()
if response:
outputBody = getattr(response, 'outputBody', None)
if outputBody is not None:
outputBody()
else:
response = str(response)
if response:
stdout.write(response)
# The module defined a post-access function, call it
if after_list[0] is not None:
after_list[0]()
finally:
if request is not None:
request.close()
clearRequest()
if must_die:
# Try to turn exception value into an exit code.
try:
if hasattr(must_die[1], 'code'):
code = must_die[1].code
else:
code = int(must_die[1])
except:
code = must_die[1] and 1 or 0
if hasattr(request.response, '_requestShutdown'):
request.response._requestShutdown(code)
try:
reraise(must_die[0], must_die[1], must_die[2])
finally:
must_die = None
return status
_l = allocate_lock()
def get_module_info(module_name, modules={},
acquire=_l.acquire,
release=_l.release):
if module_name in modules:
return modules[module_name]
if module_name[-4:] == '.cgi':
module_name = module_name[:-4]
acquire()
tb = None
g = globals()
try:
try:
module = __import__(module_name, g, g, ('__doc__',))
# Let the app specify a realm
if hasattr(module, '__bobo_realm__'):
realm = module.__bobo_realm__
elif _default_realm is not None:
realm = _default_realm
else:
realm = module_name
# Check for debug mode
if hasattr(module, '__bobo_debug_mode__'):
debug_mode = bool(module.__bobo_debug_mode__)
else:
debug_mode = _default_debug_mode
bobo_before = getattr(module, "__bobo_before__", None)
bobo_after = getattr(module, "__bobo_after__", None)
if hasattr(module, 'bobo_application'):
object = module.bobo_application
elif hasattr(module, 'web_objects'):
object = module.web_objects
else:
object = module
error_hook = getattr(module, 'zpublisher_exception_hook', None)
validated_hook = getattr(
module, 'zpublisher_validated_hook', validate_user)
transactions_manager = getattr(
module, 'zpublisher_transactions_manager', None)
if not transactions_manager:
# Create a default transactions manager for use
# by software that uses ZPublisher and ZODB but
# not the rest of Zope.
transactions_manager = DefaultTransactionsManager()
info = (bobo_before, bobo_after, object, realm, debug_mode,
error_hook, validated_hook, transactions_manager)
modules[module_name] = modules[module_name + '.cgi'] = info
return info
except Exception:
t, v, tb = sys.exc_info()
reraise(t, str(v), tb)
finally:
tb = None
release()
class DefaultTransactionsManager(object):
def begin(self):
transaction.begin()
def commit(self):
if transaction.isDoomed():
transaction.abort()
else:
transaction.commit()
def abort(self):
transaction.abort()
def publish_module(module_name,
stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr,
environ=os.environ, debug=0, request=None, response=None):
""" publish a Python module """
return publish_module_standard(module_name, stdin, stdout, stderr,
environ, debug, request, response)
I noticed a couple of anomalies in your code.
The first is that you're creating a transaction inside of a function you have decorated with capture_span:
@elasticapm.capture_span()
def publish_module_standard(
...
client.begin_transaction('request')
A span can only be captured inside a running transaction. So I'm wondering if you're creating duplicate transactions or something, or if you should just remove that capture_span() decorator.
Additionally, the only place you're capturing errors is in this single function call:
response = publish(request, module_name, after_list, debug=debug)
And inside of this function call you have a very broad try:except: (with a bare except: clause) which doesn't raise exceptions it catches. In fact, you have a raise in your else clause! In a try:except:else: clause, the else is only executed if there was no exception, so your raise doesn't do anything.
Almost all errors will be caught by your try:except: in your publish() function and will never be seen by the outer elasticapm.capture_exception() block.
hi @basepi ,
Thanks for your suggestion . Removed capture_span() decorator.
What would be the ideal way to capture the exception and display(send the details to APM Server) it in ELASTIC APM UI .
Here I have the code and above as well. Can you help me here please
########################### Elastic APM ####################
client.begin_transaction('request')
#try:
response = publish(request, module_name, after_list, debug=debug)
#except Exception:
#message = traceback.format_exc()
#client.capture_exception()
#raise
#import pdb; pdb.set_trace()
#elasticapm.set_context(lambda: get_data_from_request(request), "request")
#elasticapm.set_context(lambda: get_data_from_response(response), "response")
#finally:
client.end_transaction(request.method + " " + (request["ACTUAL_URL"]).split("/",3)[3], response.status)
#client.end_transaction()
except (SystemExit, ImportError):
# XXX: Rendered ImportErrors were never caught here because they
# were re-raised as string exceptions. Maybe we should handle
# ImportErrors like all other exceptions. Currently they are not
# re-raised at all, so they don't show up here.
must_die = sys.exc_info()
request.response.exception(1)
except:
# debug is just used by tests (has nothing to do with debug_mode!)
if debug:
raise
request.response.exception()
status = response.getStatus()
Please Advice , Thank you!!
As I noted before, the problem is in the publish() function. The errors are not reaching the capture_exception() call because you're catching them in the try:except: in the publish() function. You either need to put the capture_exception() call inside the publish() try:except, or you need to properly handle exceptions and re-raise them so the outer call sees them.
Hi @basepi,
Please post the code here. I would like to try both.
Am not getting how to interpret your statements.
Just need help with the snippet.
Help me as am in the last stages of POC
If you just add an elasticapm.capture_exception() call to your except: block inside of your publish() function, it may fix most of your problem:
except:
# save in order to give 'PubFailure' the original exception info
exc_info = sys.exc_info()
elasticapm.capture_exception() # this is the new line, l130 in your original paste
Hi @basepi
How to populate error rate graph on the overview window in APM ui , not getting though ?
Thanks
hi @basepi , @lwintergerst
We are getting the "Request Traced" for MEMCACHE, how do I get "STACK TRACE" for all the MEMCACHE related calls.
Currently we observe - Client.gets is being logged/displayed in the UI, no further details though.
How do we make "Client.gets" to have more information ? like what is being sent to memcache
Appreciate your help here @basepi @lwintergerst
We don't collect stack traces by default on spans shorter than 5ms. This is because collecting the stack trace does cause some overhead, especially when there are many very short spans.
You can change this behavior with the span_stack_trace_min_duration config.
As far as naming of the spans goes, we want the span names to be low-cardinality so we don't include query information in the name. We don't currently collect the query in the metadata for the span -- nobody has asked for that information before. 
If you wanted to open an issue, we could add that to our backlog. You could make a PR to add that context information to the span. See the existing instrumentation (the linked one is for python_memcached, there's a different instrumentation for pymemcache) and add db.statement as shown here in the dbapi2 instrumentation.
Hi @basepi,
Let me open an issue and see how feasible it would be to get pymemcached metadata in the span names
Thank you
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
