Request http error on Wildfly 9 io.undertow.servlet.spec.HttpServletRequestImpl

Elastic Observability APM
,
1

Kibana version: 8.6.2

Elasticsearch version: 8.6.2

APM Server version: 8.6.2

APM Agent language and version: Java on 1.36.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker compose into docker swarm cluster

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup?
I'm using traefik with load balancer in front of all solution. RUM running perfectly and others applications too.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Hi everyone!

I'm trying to run apm in my test environment, but I received errors when make some requests to the application with apm enabled.

These errors don't occur when APM is disabled. Maybe, I made some mistakes in configuration, but I really need help to find a solution.

I tried to change the agent configuration to disable headers capture and others features to try a solution but now I have no idea how to resolve it.

I'm running Wildfly 9.0.2 with Elastic APM 1.36.0 on Java 1.8.0_302 Runtime version: 1.8.0_302-b08 VM version: 25.302-b08 (IcedTea) Linux 5.4.235-144.344.amzn2.x86_64.

This is the startup configuration:

JAVA_OPTS:  -XX:+UseCompressedOops  -XX:+UseCompressedOops -javaagent:/usr/local/apm/elastic-apm-agent.jar
-Delastic.apm.service_name=api-saude-teste-miguel
-Delastic.apm.server_urls=https://apm.betha.services
-Delastic.apm.secret_token=LKJSHDFLKJSDHFLSDKJFHakdjfshlsdkjfh
-Delastic.apm.environment=test
-Delastic.apm.application_packages=com.betha.saude
-Delastic.apm.include_process_args=false
-Delastic.apm.global_labels=app=api-saude,vertical=saude,startup=saude
-Delastic.apm.capture_headers=false
-Delastic.apm.use_path_as_transaction_name=false
-Delastic.apm.jms_listener_packages=
-Delastic.apm.transaction_ignore_urls=/metrics,/prometheus,*/metrics,*/health,*/health/*,*/prometheus,*/prometheus/*,*/actuator,*/actuator/*,*/alive,*/alive/*
-Delastic.apm.use_jaxrs_path_as_transaction_name=true
-Delastic.apm.log_level=DEBUG
 -Xbootclasspath/p:/opt/jboss/wildfly/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.0.Final.jar -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/opt/jboss/wildfly/javaagent/jmx_prometheus
/jmx_prometheus.jar=9103:/opt/jboss/wildfly/javaagent/jmx_prometheus/wildfly.yaml -Xms4096m -Xmx6272m -XX:NewSize=1024m -XX:MaxNewSize=1024m -XX:MaxMetaspaceSize=520m -XX:CompressedClassSpaceSize=70m -XX:ReservedCodeCacheSize=130m -XX:Min
HeapFreeRatio=40 -XX:MaxHeapFreeRatio=80 -XX:CMSInitiatingOccupancyFraction=90 -server -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ScavengeBefore
FullGC -XX:+CMSScavengeBeforeRemark -XX:+CMSClassUnloadingEnabled -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+UseCompressedOops -XX:+ParallelRefProcEnabled -Xss256k -Djava.net.preferIPv4Stack=true -Duser.country=BR -Duser.langu
age=pt -Duser.timezone=America/Sao_Paulo -Djboss.modules.system.pkgs=org.jboss.byteman,com.manageengine,co.elastic,org.jboss.logmanager -Djava.awt.headless=true -Doauth.noValidationToken=true -Doauth.host=http://plataforma-oauth.test..../auth -Dfile.encoding=UTF-8 -Xss512k -XX:+CMSClassUnloadingEnabled -XX:+UnlockDiagnosticVMOptions

I enabled log debug like startup show, and this is a part of stack trace with error:

2023-04-06 15:26:00,108 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 606)
2023-04-06 15:26:00,109 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /usuarios/listByEntity' 00-f5f1014fcec1926edd7cdb8a7515a94b-036240d9c053e27a-01 (401947c) (3)
2023-04-06 15:26:00,110 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'EVAL' 00-f5f1014fcec1926edd7cdb8a7515a94b-cd31f9546093a16c-01 (4d023297) (0)
2023-04-06 15:26:00,111 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 607)
2023-04-06 15:26:00,114 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /usuarios/listByEntity' 00-f5f1014fcec1926edd7cdb8a7515a94b-036240d9c053e27a-01 (401947c) (2)
2023-04-06 15:26:00,116 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET plataforma-auth.test...' 00-f5f1014fcec1926edd7cdb8a7515a94b-323884f4884d8ae5-01 (113a8708) (0)
15:26:00,123 ERROR [org.jboss.as.ejb3.invocation] (default task-7) WFLYEJB0034: EJB Invocation failed on component UsuarioResource for method public javax.ws.rs.core.Response com.betha.saude.api.usuario.UsuarioResource.listByEntity(int,int,java.lang.String,java.lang.String): javax.ejb.EJBException: java.lang.ClassCastException: io.undertow.servlet.spec.HttpServletRequestImpl cannot be cast to java.util.Enumeration
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:213)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:265)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
  at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:634)
  at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
  at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
  at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
  at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
  at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
  at com.betha.saude.api.usuario.UsuarioResource$$$view383.listByEntity(Unknown Source)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
  at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
  at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
  at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)
  at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
  at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
  at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
  at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
  at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
  at com.betha.bfc.rest.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:51)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.bfc.commons.bfc.security.constraint.UserPermissionFilter.doFilter(UserPermissionFilter.java:70)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.bfc.security.UserAccessEntityHashFilter.doFilter(UserAccessEntityHashFilter.java:78)
  at com.betha.bfc.security.UserAccessAttributesFilter.doFilter(UserAccessAttributesFilter.java:37)
  at com.betha.bfc.security.UserAccessFilter.doFilter(UserAccessFilter.java:29)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.oauth2.token.AccessTokenValidatorStandard.doFilter(AccessTokenValidatorStandard.java:48)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at org.ebaysf.web.cors.CORSFilter.handleSimpleCORS(CORSFilter.java:302)
  at org.ebaysf.web.cors.CORSFilter.doFilter(CORSFilter.java:160)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.saude.enterprise.StaleClientAppFilter.doFilter(StaleClientAppFilter.java:23)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
  at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
  at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: io.undertow.servlet.spec.HttpServletRequestImpl cannot be cast to java.util.Enumeration
  at org.jboss.weld.proxies.HttpServletRequest$894284377$Proxy$_$$_WeldClientProxy.getHeaders(Unknown Source)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService.getAppContext(UsuarioAccessService.java:56)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService.lambda$getApiByEntity$1(UsuarioAccessService.java:92)
  at feign.SynchronousMethodHandler.targetRequest(SynchronousMethodHandler.java:154)
  at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:88)
  at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
  at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
  at com.betha.saude.enterprise.usuarios.$Proxy226.getUsuariosAccessByEntity(Unknown Source)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService.getUsuariosAccessByEntity(UsuarioAccessService.java:69)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService$Proxy$_$$_WeldClientProxy.getUsuariosAccessByEntity(Unknown Source)
  at com.betha.saude.api.usuario.UsuarioResource.listByEntity(UsuarioResource.java:52)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
  at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
  at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
  at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
  at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
  ... 99 more

15:26:00,139 ERROR [com.betha.bfc.rest.jaxrs.ApplicationExceptionMapper] (default task-7) ApiError(message=O servidor apresentou um erro interno ao processar a requisição.): java.lang.ClassCastException: io.undertow.servlet.spec.HttpServletRequestImpl cannot be cast to java.util.Enumeration
  at org.jboss.weld.proxies.HttpServletRequest$894284377$Proxy$_$$_WeldClientProxy.getHeaders(Unknown Source)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService.getAppContext(UsuarioAccessService.java:56)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService.lambda$getApiByEntity$1(UsuarioAccessService.java:92)
  at feign.SynchronousMethodHandler.targetRequest(SynchronousMethodHandler.java:154)
  at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:88)
  at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
  at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
  at com.betha.saude.enterprise.usuarios.$Proxy226.getUsuariosAccessByEntity(Unknown Source)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService.getUsuariosAccessByEntity(UsuarioAccessService.java:69)
  at com.betha.saude.enterprise.usuarios.UsuarioAccessService$Proxy$_$$_WeldClientProxy.getUsuariosAccessByEntity(Unknown Source)
  at com.betha.saude.api.usuario.UsuarioResource.listByEntity(UsuarioResource.java:52)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
  at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
  at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
  at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
  at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
  at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:634)
  at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
  at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
  at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
  at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
  at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
  at com.betha.saude.api.usuario.UsuarioResource$$$view383.listByEntity(Unknown Source)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
  at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
  at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
  at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)
  at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
  at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
  at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
  at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
  at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
  at com.betha.bfc.rest.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:51)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.bfc.commons.bfc.security.constraint.UserPermissionFilter.doFilter(UserPermissionFilter.java:70)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.bfc.security.UserAccessEntityHashFilter.doFilter(UserAccessEntityHashFilter.java:78)
  at com.betha.bfc.security.UserAccessAttributesFilter.doFilter(UserAccessAttributesFilter.java:37)
  at com.betha.bfc.security.UserAccessFilter.doFilter(UserAccessFilter.java:29)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.oauth2.token.AccessTokenValidatorStandard.doFilter(AccessTokenValidatorStandard.java:48)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at org.ebaysf.web.cors.CORSFilter.handleSimpleCORS(CORSFilter.java:302)
  at org.ebaysf.web.cors.CORSFilter.doFilter(CORSFilter.java:160)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at com.betha.saude.enterprise.StaleClientAppFilter.doFilter(StaleClientAppFilter.java:23)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
  at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
  at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)


2023-04-06 15:26:00,141 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving ERROR event (sequence 608)
2023-04-06 15:26:00,144 [default task-7] DEBUG co.elastic.apm.agent.impl.ActiveStack - Deactivating 'GET /usuarios/listByEntity' 00-f5f1014fcec1926edd7cdb8a7515a94b-036240d9c053e27a-01 (401947c) on thread 237
2023-04-06 15:26:00,144 [default task-7] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /usuarios/listByEntity' 00-f5f1014fcec1926edd7cdb8a7515a94b-036240d9c053e27a-01 (401947c) (1)
2023-04-06 15:26:00,145 [default task-7] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endTransaction 'GET /usuarios/listByEntity' 00-f5f1014fcec1926edd7cdb8a7515a94b-036240d9c053e27a-01 (401947c)
2023-04-06 15:26:00,147 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 609)
2023-04-06 15:26:00,147 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /usuarios/listByEntity' 00-f5f1014fcec1926edd7cdb8a7515a94b-036240d9c053e27a-01 (401947c) (0)
2023-04-06 15:26:00,736 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening https://apm.../config/v1/agents without proxy
2

Hi @miguel.longo , welcome to our forum and thanks or reporting this.

While the agent might indirectly trigger the issue, the provided stack trace does not directly involve the agent code, so here the agent might have a side-effect on the application. Maybe there is another clue in the complete agent log (setting log_file to store it in a separate file is usually a good idea to help debugging).

A very simple way to avoid this kind of issues is to make the agent start with a small delay, thus attempting to avoid side-effects during application startup.

Could you try with adding -Delastic.apm.delay_agent_premain_ms=10000 to your JVM parameters ?
This will make the agent start after waiting 10 seconds, thus allowing the application startup to finish without side effects, you'll have to adapt to your actual application startup time if it takes longer. Also, as class-loading only happens lazily, if you can trigger some activity on the EJB invocation before the agent starts, that would be even better.

Hopefully that will provide us a work-around here, and then we can investigate further what is happening and why the agent might trigger this.

3

Hi @Sylvain_Juge, thank you for your reply.

I will try to apply the conf in JVM like your comment and return here if I have another question/issue.

Thank you again.

5

Hi @Sylvain_Juge.

I did the tests with the delay parameters, even so I get the error in ClassCastException.

Is there something regarding version compatibility? could you tell me where else i can look?

Thank you.

6

I would suspect that this might be a side effect of an instrumentation, though the stacktrace gives us no idea which one it could be.

I would suggest you to try the following steps to pinpoint the issue:

  • Start the agent with all instrumentation disabled (config option). Check whether your error still occurs. If it doesn't occur anymore, we can start trying to pinpoint the exact instrumentation using the following steps.
  • With instrumentation enabled, the agent prints a log statement Used instrumentation groups: XYZ on shutdown. This tells us, which instrumentations where actually used by your application.
  • Try to find the exact instrumentations causing the issue by tuning the disable_instrumentations based on the actually used instrumentations
7

Hi Jonas.

We verified that the agent instruments ThreadPoolExecutors by default and that possibly this was the problem.

We relied on this issue: Bytecode instrumentation does not respect ClassLoader · Issue #1606 · elastic/apm-agent-java · GitHub

In the logs we managed to find the two services, the second being the class with the same location that appears in the errors, the _WeldClientProxy:

elastic-agent
elastic-agent1897×205 202 KB

This is a part of the error log:
at org.jboss.weld.proxies.HttpServletRequest$894284377$Proxy$_$$_WeldClientProxy.getHeaders(Unknown Source)

I don't know if it was a coincidence of WeldClientProxy, but to test it, we used the argument -Delastic.apm.classes_excluded_from_instrumentation=org.jboss.* to have only ThreadPoolExecutors from elastic, everything worked perfectly both in the application and in apm.

8

Great find with the linked issue! However, that specific root cause should be fixed in the APM Agent version you are using (1.36.0) unless there is a regression. It would be great if you could help us narrow it down.

I don't think that your specific problem is caused by the executor instrumentation, but maybe by our servlet-API instrumentation.

Could you try the following steps:

  • change the classes_excluded_from_instrumentation to org.jboss.weld.proxies.HttpServletRequest* to verify that actually an instrumentation of that proxy-class is causing the issue
  • try without the classes_excluded_from_instrumentation but with -Delastic.apm.enable_type_pool_cache=false. If the problem does not occur, we might indeed have a regression of the linked issue.
9

Hi Jonas.

In my tests, with -Delastic.apm.enable_type_pool_cache=false, all errors occur again.

With -Delastic.apm.classes_excluded_from_instrumentation=org.jboss.weld.proxies.HttpServletRequest*, all works fine without problems.

I used agents in versions 1.36.0 to 1.38.0, and got the same behavior with two arguments.

I've been using the agent on version 1.37.0 with argument for a few days and I haven't had any issues.

Thank you, Jonas.

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.