Elastic APM exception when instrumenting Java CXF application

Elastic Observability APM
1

Elasticsearch version:
7.6

APM Server version:
8.0.0

APM Agent language and version:
Java agent version: 1.17.0

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
We see the below stack trace when sending http request using apache cxf.
We see elastic APM file SpanInScopeRunnableWrapper.java in the stack trace.
Could you pls suggest what could be reason for this exception.
(Have removed few repeated stacktraces due to size limit)
Provide logs and/or server output (if relevant):

{
  "stacktrace": [
    {
      "function": "process",
      "library_frame": true,
      "module": "com.asg.iconductor.processor",
      "filename": "ResponseQueueMsgProcessor.java",
      "classname": "com.asg.iconductor.processor.ResponseQueueMsgProcessor",
      "exclude_from_grouping": false,
      "line": {
        "number": 54
      }
    },
    {
      "function": "process",
      "library_frame": true,
      "module": "org.apache.camel.processor",
      "filename": "DelegateSyncProcessor.java",
      "classname": "org.apache.camel.processor.DelegateSyncProcessor",
      "exclude_from_grouping": false,
      "line": {
        "number": 63
      }
    },
    {
      "function": "process",
      "library_frame": true,
      "module": "org.apache.camel.processor",
      "filename": "DelegateAsyncProcessor.java",
      "classname": "org.apache.camel.processor.DelegateAsyncProcessor",
      "exclude_from_grouping": false,
      "line": {
        "number": 110
      }
    },
    {
      "function": "process",
      "library_frame": true,
      "module": "org.apache.camel.processor",
      "filename": "RedeliveryErrorHandler.java",
      "classname": "org.apache.camel.processor.RedeliveryErrorHandler",
      "exclude_from_grouping": false,
      "line": {
        "number": 548
      }
    },
    {
      "function": "process",
      "library_frame": true,
      "module": "org.apache.camel.processor",
      "filename": "CamelInternalProcessor.java",
      "classname": "org.apache.camel.processor.CamelInternalProcessor",
      "exclude_from_grouping": false,
      "line": {
        "number": 201
      }
    },
    {
      "function": "process",
      "exclude_from_grouping": false,
      "module": "org.apache.camel.processor",
      "filename": "Pipeline.java",
      "classname": "org.apache.camel.processor.Pipeline",
      "library_frame": true,
      "line": {
        "number": 138
      }
    },
    {
      "function": "process",
      "library_frame": true,
      "module": "org.apache.camel.processor",
      "filename": "Pipeline.java",
      "classname": "org.apache.camel.processor.Pipeline",
      "exclude_from_grouping": false,
      "line": {
        "number": 101
      }
    },
    {
      "function": "process",
      "library_frame": true,
      "module": "org.apache.camel.processor",
      "filename": "CamelInternalProcessor.java",
      "classname": "org.apache.camel.processor.CamelInternalProcessor",
      "exclude_from_grouping": false,
      "line": {
        "number": 201
      }
    },
    {
      "function": "asyncInvoke",
      "module": "org.apache.camel.component.cxf",
      "exclude_from_grouping": false,
      "filename": "CxfConsumer.java",
      "classname": "org.apache.camel.component.cxf.CxfConsumer$CxfConsumerInvoker",
      "library_frame": true,
      "line": {
        "number": 206
      }
    },
    {
      "function": "invoke",
      "library_frame": true,
      "module": "org.apache.camel.component.cxf",
      "filename": "CxfConsumer.java",
      "classname": "org.apache.camel.component.cxf.CxfConsumer$CxfConsumerInvoker",
      "exclude_from_grouping": false,
      "line": {
        "number": 182
      }
    },
    {
      "function": "run",
      "library_frame": true,
      "module": "org.apache.cxf.interceptor",
      "filename": "ServiceInvokerInterceptor.java",
      "classname": "org.apache.cxf.interceptor.ServiceInvokerInterceptor$1",
      "exclude_from_grouping": false,
      "line": {
        "number": 59
      }
    },
    {
      "function": "call",
      "library_frame": true,
      "module": "java.util.concurrent",
      "filename": "Executors.java",
      "classname": "java.util.concurrent.Executors$RunnableAdapter",
      "exclude_from_grouping": false,
      "line": {
        "number": 511
      }
    },
    {
      "function": "run",
      "library_frame": true,
      "module": "java.util.concurrent",
      "filename": "FutureTask.java",
      "classname": "java.util.concurrent.FutureTask",
      "exclude_from_grouping": false,
      "line": {
        "number": 266
      }
    },
    {
      "function": "run",
      "module": "org.apache.cxf.interceptor",
      "exclude_from_grouping": false,
      "filename": "ServiceInvokerInterceptor.java",
      "classname": "org.apache.cxf.interceptor.ServiceInvokerInterceptor$2",
      "library_frame": true,
      "line": {
        "number": 126
      }
    },
    {
      "function": "run",
      "library_frame": true,
      "module": "co.elastic.apm.agent.impl.async",
      "filename": "SpanInScopeRunnableWrapper.java",
      "classname": "co.elastic.apm.agent.impl.async.SpanInScopeRunnableWrapper",
      "exclude_from_grouping": false,
      "line": {
        "number": 64
      }
    },
    {
      "function": "execute",
      "library_frame": true,
      "module": "org.apache.cxf.workqueue",
      "filename": "SynchronousExecutor.java",
      "classname": "org.apache.cxf.workqueue.SynchronousExecutor",
      "exclude_from_grouping": false,
      "line": {
        "number": 37
      }
    },
    {
      "function": "handleMessage",
      "library_frame": true,
      "module": "org.apache.cxf.interceptor",
      "filename": "ServiceInvokerInterceptor.java",
      "classname": "org.apache.cxf.interceptor.ServiceInvokerInterceptor",
      "exclude_from_grouping": false,
      "line": {
        "number": 131
      }
    },
    {
      "function": "doIntercept",
      "library_frame": true,
      "module": "org.apache.cxf.phase",
      "filename": "PhaseInterceptorChain.java",
      "classname": "org.apache.cxf.phase.PhaseInterceptorChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 309
      }
    },
    {
      "function": "onMessage",
      "library_frame": true,
      "module": "org.apache.cxf.transport",
      "filename": "ChainInitiationObserver.java",
      "classname": "org.apache.cxf.transport.ChainInitiationObserver",
      "exclude_from_grouping": false,
      "line": {
        "number": 121
      }
    },
    {
      "function": "invoke",
      "library_frame": true,
      "module": "org.apache.cxf.transport.http",
      "filename": "AbstractHTTPDestination.java",
      "classname": "org.apache.cxf.transport.http.AbstractHTTPDestination",
      "exclude_from_grouping": false,
      "line": {
        "number": 262
      }
    },
    {
      "function": "invokeDestination",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "ServletController.java",
      "classname": "org.apache.cxf.transport.servlet.ServletController",
      "exclude_from_grouping": false,
      "line": {
        "number": 234
      }
    },
    {
      "function": "invoke",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "ServletController.java",
      "classname": "org.apache.cxf.transport.servlet.ServletController",
      "exclude_from_grouping": false,
      "line": {
        "number": 208
      }
    },
    {
      "function": "invoke",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "ServletController.java",
      "classname": "org.apache.cxf.transport.servlet.ServletController",
      "exclude_from_grouping": false,
      "line": {
        "number": 160
      }
    },
    {
      "function": "invoke",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "CXFNonSpringServlet.java",
      "classname": "org.apache.cxf.transport.servlet.CXFNonSpringServlet",
      "exclude_from_grouping": false,
      "line": {
        "number": 180
      }
    },
    {
      "function": "handleRequest",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "AbstractHTTPServlet.java",
      "classname": "org.apache.cxf.transport.servlet.AbstractHTTPServlet",
      "exclude_from_grouping": false,
      "line": {
        "number": 299
      }
    },
    {
      "function": "doPost",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "AbstractHTTPServlet.java",
      "classname": "org.apache.cxf.transport.servlet.AbstractHTTPServlet",
      "exclude_from_grouping": false,
      "line": {
        "number": 218
      }
    },
    {
      "function": "service",
      "module": "javax.servlet.http",
      "exclude_from_grouping": false,
      "filename": "HttpServlet.java",
      "classname": "javax.servlet.http.HttpServlet",
      "library_frame": true,
      "line": {
        "number": 648
      }
    },
    {
      "function": "service",
      "library_frame": true,
      "module": "org.apache.cxf.transport.servlet",
      "filename": "AbstractHTTPServlet.java",
      "classname": "org.apache.cxf.transport.servlet.AbstractHTTPServlet",
      "exclude_from_grouping": false,
      "line": {
        "number": 274
      }
    },
    {
      "function": "internalDoFilter",
      "library_frame": true,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 292
      }
    },
    {
      "function": "doFilter",
      "library_frame": true,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 207
      }
    },
    {
      "function": "internalDoFilter",
      "library_frame": true,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 240
      }
    },
    {
      "function": "doFilter",
      "library_frame": true,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 207
      }
    },
    {
      "function": "doFilterInternal",
      "module": "org.springframework.boot.web.filter",
      "exclude_from_grouping": false,
      "filename": "ApplicationContextHeaderFilter.java",
      "classname": "org.springframework.boot.web.filter.ApplicationContextHeaderFilter",
      "library_frame": true,
      "line": {
        "number": 55
      }
    },
    {
      "function": "doFilter",
      "library_frame": true,
      "module": "org.springframework.web.filter",
      "filename": "OncePerRequestFilter.java",
      "classname": "org.springframework.web.filter.OncePerRequestFilter",
      "exclude_from_grouping": false,
      "line": {
        "number": 107
      }
    },
   {
      "function": "doFilter",
      "exclude_from_grouping": false,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "library_frame": true,
      "line": {
        "number": 207
      }
    },
   {
      "function": "internalDoFilter",
      "library_frame": true,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 240
      }
    },
  {
      "function": "doFilter",
      "library_frame": true,
      "module": "org.apache.catalina.core",
      "filename": "ApplicationFilterChain.java",
      "classname": "org.apache.catalina.core.ApplicationFilterChain",
      "exclude_from_grouping": false,
      "line": {
        "number": 207
      }
    },
    {
      "function": "doFilterInternal",
      "library_frame": true,
      "module": "org.springframework.web.filter",
      "filename": "HttpPutFormContentFilter.java",
      "classname": "org.springframework.web.filter.HttpPutFormContentFilter",
      "exclude_from_grouping": false,
      "line": {
        "number": 109
      }
    }
  ],
  "message": "java.lang.LinkageError cannot be cast to java.lang.Exception",
  "type": "java.lang.Exception"
}
2

Hi Priya and welcome to the forum :wave:

SpanInScopeRunnableWrapper is just a wrapper of the original Runnable that was submitted for execution by the SynchronousExecutor. Besides creating a span to capture the async execution, it merely delegates the run invocation to the original Runnable and not a lot more. So, it is likely that the exception is related to the actual execution of the submitted task (and thanks to our wrapper you get this exception and have it linked to the rest of the trace from the submitting thread).

If indeed you suspect that the usage of SpanInScopeRunnableWrapper is the cause for the error, upgrade your agent to the latest version, where the instrumentation of Executors have been replaced with a new one that doesn't wrap Runnables (unless they are used as Lambdas).

3

Thanks a lot for your response.
Yes, we will try with the latest version.

Also, regarding the point
"So, it is likely that the exception is related to the actual execution of the submitted task" - the application works fine when we do not attach the agent. That's why we were suspecting the issue could be with elastic apm.
Thanks,
Priya

4

If the application doesn't work as expected with the agent, this is definitely something to look at. I understood you only saw this stack trace in the APM UI, which may be the case if this Exception, thrown on a worker thread forked from the main thread regardless of the agent, is otherwise swallowed quietly or ignored.

5

Thanks for the response.
I also missed to mention in my earlier messages that we see the below error on agent startup
2020-09-14 19:00:52,934 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpClientInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpClientInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

Could the exception that we are seeing be related to this error.
Please help.

6

This is not a real error, it is expected when running on Java<11. We addressed that in Eliminate Java-11 HttpClient plugin class version errors by eyalkoren · Pull Request #1397 · elastic/apm-agent-java · GitHub.

As far as I can tell, the two reported issues should not coexist - the log message above means you are using agent version 1.18.0, in which SpanInScopeRunnableWrapper should not exist anymore.

If you experience a difference in the behaviour of the application itself with and without the agent - please report that.

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.