Not all errors/exceptions are reported by the Java APM agent

Elastic Observability APM
#1

I set up the apm Java agent with Tomcat server, but I notice there are some transactions(Http request) that return 5xx errors but they don't have exception stack traces(I couldn't find them in APM Errors by searching the transaction id). The sampling rate of the agent is set to 100%. Could you briefly explain the following questions? Thanks!

  • How are errors tracked by the agent?
  • Why do some 5xx requests not have error traces?
  • If the sampling rate is 100%, should all error traces be

OS version : CentOS Linux, Version 7(core)
Java Version : openjdk version "1.8.0_275"
JVM Setting : default GC setting with Parallel GC, maximum JVM size is 6GB
APM java agent version : 1.19.0

APM java agent configuration
The APM agent is setup with the Tomcat process with -javaagent flag

/java/1.8.0_275/bin/java -javaagent:elastic-apm-agent-1.19.0.jar -Delastic.apm.config_file=elastic_apm.properties -Delastic.apm.log_file=elastic-apm.log -Delastic.apm.service_name=web-java
#2

Hi @bbking,

First, we need to distinguish two things:

  • HTTP 5xx status code for HTTP responses that indicate a server error
  • Exceptions that may be thrown by application code and are reported as Errors in Kibana

Those two things are captured separately, thus having one of them does not imply the other, which means you can have 5xx + Error, 5xx without Error, Error without 5xx or even multiple Errors reported for a single transaction.

Agent instruments methods by wrapping their body, and captures exceptions in a similar way as the following pseudo-code:

Original method before instrumentation:

void methodToInstrument() {
  // method body, may throw any Exception
}

Instrumented method

void methodToInstrument() {
  try {
    // method body, may throw any Exception
  } catch (Exception thrown){
    // report exception as an Error
   throw thrown; // rethrow the original exception to preserve original behavior
  }
}

Thus, that means that:

  • exceptions that are thrown and caught within method body (or below) aren't reported as errors
  • exceptions that are thrown in the calling code that calls methodToInstrument() are not captured as errors (for example within internal application server request processing that calls the instrumented method).
  • exceptions that are thrown and not caught within method body are reported as errors, aka "the ones that escape the method body"

In your case, you need to identify what transactions are triggering 5xx responses without having errors:

  • is there any stack trace visible in tomcat log for those ?
  • is there any servlet filter that wraps execution in a try/catch block, which is common for logging ?
#3

Thanks for the explanations! We use struts on Tomcat and exceptions are caught and wrapped in a struts interceptor. I guess struts code are not instrumented and that's why some exceptions are not recorded by the agent.