ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!

miguel.longo · July 4, 2023, 7:33pm

Kibana version: 8.6.0

Elasticsearch version: 8.6.0

APM Server version: 8.6.0

APM Agent language and version: 1.36.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker stack from registry

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? Nothing in specific

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I'm running a java wildfly application with elastic apm agent and I got no transactions in APM, only errors are showing in error panel.

Searching in discovery, filtering to not show errors, I can see some transactions with span id and transaction id, but I believe it is not complete, apparently.

I enabled debug to try to see more details and I got it in log:

[INFO ]2023-07-03 17:38:47,684 [default task-11] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'POST /omitted/{id}' 00-71dded60c426f650e284369cf7e28ad5-0351617688689cfd-01 (3899fe1c) (6)
[INFO ]2023-07-03 17:38:47,685 [default task-11] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'KafkaProducer#send to omitted' 00-71dded60c426f650e284369cf7e28ad5-0586c9d4c5b85112-01 (4162bd52) (1)
17:38:47,686 INFO  [stdout] (default task-7) 2023-07-03 17:38:47,686 [default task-7] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Empty servlet path fallback applied. requestURI = /omitted/40358242, contextPath = /omitted/, servletPath = , pathInfo = /omitted/40358242
17:38:47,686 INFO  [stdout] (default task-7) 2023-07-03 17:38:47,686 [default task-7] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
17:38:47,686 INFO  [stdout] (default task-7) java.lang.StringIndexOutOfBoundsException: String index out of range: -1
17:38:47,686 INFO  [stdout] (default task-7) 	at java.lang.String.substring(String.java:1967) ~[?:1.8.0_302]
17:38:47,686 INFO  [stdout] (default task-7) 	at co.elastic.apm.agent.servlet.ServletTransactionHelper.normalizeServletPath(ServletTransactionHelper.java:216) ~[elastic-apm-agent.jar:1.37.0]
17:38:47,686 INFO  [stdout] (default task-7) 	at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:249) ~[elastic-apm-agent.jar:1.37.0]
17:38:47,686 INFO  [stdout] (default task-7) 	at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent.jar:1.37.0]
17:38:47,686 INFO  [stdout] (default task-7) 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,686 INFO  [stdout] (default task-7) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
17:38:47,687 INFO  [stdout] (default task-7) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
17:38:47,687 INFO  [stdout] (default task-7) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
17:38:47,687 INFO  [stdout] (default task-7) 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
17:38:47,691 INFO  [stdout] (elastic-apm-server-reporter) 2023-07-03 17:38:47,691 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving ERROR event (sequence 183)

This error apparently is on java agent and in all my tests, I can't to solve it.

This is my agent running configuration:

-javaagent:/elastic-apm-agent.jar
	-Delastic.apm.service_name=api-omitted
	-Delastic.apm.server_urls=https://abc.apm.xxx
	-Delastic.apm.secret_token=omitted
	-Delastic.apm.environment=production
	-Delastic.apm.application_packages=omitted
	-Delastic.apm.include_process_args=true
	-Delastic.apm.global_labels=app=omitted,tier=api
	-Delastic.apm.capture_headers=true
	-Delastic.apm.use_path_as_transaction_name=true
	-Delastic.apm.transaction_ignore_urls=/metrics,/prometheus,*/metrics,*/health,*/health/*,*/prometheus,*/prometheus/*,*/actuator,*/actuator/*,*/alive,*/alive/*
	-Delastic.apm.use_jaxrs_path_as_transaction_name=true
	-Delastic.apm.circuit_breaker_enabled=false
	-Delastic.apm.classes_excluded_from_instrumentation=org.jboss.*,io.undertow.*
	-Delastic.apm.enable_type_pool_cache=false
	-Delastic.apm.central_config=true
	-Delastic.apm.log_level=ERROR

The classes_excluded_from_instrumentation argument is just here to test. The application is running with this argument now.

Is there any way to find out which class or what the problem is?

Thank you all.

Sylvain_Juge · July 5, 2023, 8:25am

Hi, thanks for reporting this @miguel.longo , this definitely looks like a corner case where there is a bug in the agent. Thanks to the provided logs I've managed to reproduce it locally thus I should be able to fix it quickly.

I'll keep you posted on the progress on fixing this.

Sylvain_Juge · July 5, 2023, 9:19am

I've found and fixed the issue in Fix servlet path normalization by SylvainJuge · Pull Request #3222 · elastic/apm-agent-java · GitHub.

Can you try to validate with this agent snapshot that the fix is effective and works as expected in your application ?

If that's ok, it means this fix could be included in the next agent release (make sure to subscribe to notifications to get notified when it's available).

miguel.longo · July 5, 2023, 12:51pm

Hi @Sylvain_Juge!

Thanks for the quick response, and sorry for the delay.

I applied the snapshot and the behavior is apparently the same.

The log shows the same error. I will send the log file directly to you.

miguel.longo · July 5, 2023, 7:50pm

Hi @Sylvain_Juge,

We fixed all the SQL errors. Now, the application is running normally and the log file I sent you inbox can be ignored.

However, java agent still prints the same errors and don't show transactions in APM.

I'm still get data from discovery from this service.

Regards.

Sylvain_Juge · July 6, 2023, 7:21am

Hi, I'm a bit confused here.

are you using the the provided snapshot agent ?
does this snapshot allows to avoid the exception you had with StringIndexOutOfBoundsException thrown from ServletTransactionHelper.normalizeServletPath method ? (if yes, then that means we have at least one minor bugfix ready to ship).

If you still don't have any transactions, then maybe removing the classes_excluded_from_instrumentation configuration could help here.

If this configuration change is not enough, then capturing the agent debug logs using this procedure is the way to go, you can still send me the log file in DM if you prefer.

miguel.longo · July 6, 2023, 12:08pm

Hi Sylvain,

are you using the the provided snapshot agent ?
Yes. In this part of log, you can see the agent version with your snapshot: ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT].

Log below.

does this snapshot allows to avoid the exception you had with StringIndexOutOfBoundsException thrown from ServletTransactionHelper.normalizeServletPath method ?
Unfortunately not. As you can see in the log below, the same "error" still happens.

18:12:15,151 INFO  [stdout] (default task-11) 2023-07-05 18:12:15,073 [default task-10] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
18:12:15,152 INFO  [stdout] (default task-11) java.lang.StringIndexOutOfBoundsException: String index out of range: -1
18:12:15,152 INFO  [stdout] (default task-11)   at java.lang.String.substring(String.java:1967) ~[?:1.8.0_302]
18:12:15,152 INFO  [stdout] (default task-11)   at co.elastic.apm.agent.servlet.ServletTransactionHelper.normalizeServletPath(ServletTransactionHelper.java:217) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
18:12:15,152 INFO  [stdout] (default task-11)   at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:248) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
18:12:15,152 INFO  [stdout] (default task-11)   at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
18:12:15,152 INFO  [stdout] (default task-11)   at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]
18:12:15,152 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,152 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,152 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,152 INFO  [stdout] (default task-11)   at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
18:12:15,152 INFO  [stdout] (default task-11)   at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,153 INFO  [stdout] (default task-11)   at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
18:12:15,154 INFO  [stdout] (default task-11)   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
18:12:15,154 INFO  [stdout] (default task-11)   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
18:12:15,154 INFO  [stdout] (default task-11)   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]

I removed the classes_excluded_from_instrumentation like you mentioned, but the behavior is the same.

I will get the trace log and send to you in DM.

Sorry for the confusion. I figured by fixing the application problem, I could possibly fix the agent problem as well, and it is not happened.

Thank you for your help and support.

Sylvain_Juge · July 6, 2023, 1:01pm

In the previous log that you provided, there was a line containing DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper which provides the parameter values that are triggering the exception in normalizeServletPath.

Could you provide the few lines that precede this stack trace in the logs ?

miguel.longo · July 6, 2023, 1:24pm

These logs?

09:36:37,102 INFO  [stdout] (default task-8) 2023-07-05 09:36:37,102 [default task-8] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Empty servlet path fallback applied. requestURI = /api-saude-outbox-process/processEvent/40890466, contextPath = /api-saude-outbox-process/, servletPath = , pathInfo = /processEvent/40890466
09:36:37,102 INFO  [stdout] (default task-10) 2023-07-05 09:36:37,102 [default task-10] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Empty servlet path fallback applied. requestURI = /api-saude-outbox-process/processEvent/40890594, contextPath = /api-saude-outbox-process/, servletPath = , pathInfo = /processEvent/40890594
09:36:37,102 INFO  [stdout] (default task-9) 2023-07-05 09:36:37,102 [default task-9] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Empty servlet path fallback applied. requestURI = /api-saude-outbox-process/processEvent/40890606, contextPath = /api-saude-outbox-process/, servletPath = , pathInfo = /processEvent/40890606
09:36:37,118 INFO  [stdout] (default task-13) 2023-07-05 09:36:37,118 [default task-13] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-61cb4b1a1f3a6f4033365ea2cfd77b07-b10404cfc411f6f4-01 (5551bad4) (1)
09:36:37,118 INFO  [stdout] (default task-13) 2023-07-05 09:36:37,118 [default task-13] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-61cb4b1a1f3a6f4033365ea2cfd77b07-b10404cfc411f6f4-01 (5551bad4)
09:36:37,118 INFO  [stdout] (default task-13) 2023-07-05 09:36:37,118 [default task-13] DEBUG co.elastic.apm.agent.impl.ActiveStack - Activating '' 00-61cb4b1a1f3a6f4033365ea2cfd77b07-b10404cfc411f6f4-01 (5551bad4) on thread 228
09:36:37,118 INFO  [stdout] (default task-13) 2023-07-05 09:36:37,118 [default task-13] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-61cb4b1a1f3a6f4033365ea2cfd77b07-b10404cfc411f6f4-01 (5551bad4) (2)
09:36:37,122 INFO  [stdout] (default task-14) 2023-07-05 09:36:37,122 [default task-14] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-17260e468f2a71f8ca9261bd2c31308b-eaba227fc892d377-01 (19d2d322) (1)
09:36:37,123 INFO  [stdout] (default task-14) 2023-07-05 09:36:37,122 [default task-14] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-17260e468f2a71f8ca9261bd2c31308b-eaba227fc892d377-01 (19d2d322)
09:36:37,123 INFO  [stdout] (default task-14) 2023-07-05 09:36:37,123 [default task-14] DEBUG co.elastic.apm.agent.impl.ActiveStack - Activating '' 00-17260e468f2a71f8ca9261bd2c31308b-eaba227fc892d377-01 (19d2d322) on thread 229
09:36:37,123 INFO  [stdout] (default task-14) 2023-07-05 09:36:37,123 [default task-14] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-17260e468f2a71f8ca9261bd2c31308b-eaba227fc892d377-01 (19d2d322) (2)
09:36:37,125 INFO  [stdout] (default task-7) 2023-07-05 09:36:37,125 [default task-7] DEBUG co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory - Creating plugin class loader for ModuleClassLoader for Module "org.jboss.log4j.logmanager:main" from local module loader @55040f2f (finder: local module finder @64c87930 (roots: /opt/jboss/wildfly/modules,/opt/jboss/wildfly/modules/system/layers/base/.overlays/layer-base-wildfly-9.0.2-Final-hibernate-validator-5.2.0-Final,/opt/jboss/wildfly/modules/system/layers/base)) containing [co.elastic.apm.agent.loginstr.AbstractLogIntegrationInstrumentation, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter$3, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter$LazyEntry, co.elastic.apm.agent.loginstr.correlation.AbstractLogCorrelationHelper$DefaultLogCorrelationHelper, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter$1, co.elastic.apm.agent.loginstr.correlation.package-info, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter$TraceIdentifierEntrySet$1, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter$2, co.elastic.apm.agent.loginstr.correlation.CorrelationIdMapAdapter$TraceIdentifierEntrySet, co.elastic.apm.agent.loginstr.package-info, co.elastic.apm.agent.loginstr.error.LoggerErrorHelper, co.elastic.apm.agent.loginstr.error.package-info, co.elastic.apm.agent.loginstr.error.AbstractLoggerErrorCapturingInstrumentation$LoggingAdvice, co.elastic.apm.agent.loginstr.error.AbstractLoggerErrorCapturingInstrumentation, co.elastic.apm.agent.loginstr.LoggingPluginClassLoaderRootPackageCustomizer, co.elastic.apm.agent.loginstr.reformatting.package-info, co.elastic.apm.agent.loginstr.reformatting.Utils, co.elastic.apm.agent.bci.classloading.LookupExposer]
09:36:37,128 INFO  [stdout] (default task-9) 2023-07-05 09:36:37,102 [default task-9] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
09:36:37,128 INFO  [stdout] (default task-9) java.lang.StringIndexOutOfBoundsException: String index out of range: -1
09:36:37,192 INFO  [stdout] (default task-9) 	at java.lang.String.substring(String.java:1967) ~[?:1.8.0_302]
09:36:37,192 INFO  [stdout] (default task-9) 	at co.elastic.apm.agent.servlet.ServletTransactionHelper.normalizeServletPath(ServletTransactionHelper.java:217) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,192 INFO  [stdout] (default task-9) 	at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:248) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,192 INFO  [stdout] (default task-9) 	at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,192 INFO  [stdout] (default task-9) 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,192 INFO  [stdout] (default task-9) 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,193 INFO  [stdout] (default task-9) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
09:36:37,193 INFO  [stdout] (default task-9) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
09:36:37,193 INFO  [stdout] (default task-9) 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
09:36:37,195 INFO  [stdout] (default task-10) 2023-07-05 09:36:37,103 [default task-8] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
09:36:37,196 INFO  [stdout] (default task-10) java.lang.StringIndexOutOfBoundsException: String index out of range: -1
09:36:37,197 INFO  [stdout] (default task-10) 	at java.lang.String.substring(String.java:1967) ~[?:1.8.0_302]
09:36:37,197 INFO  [stdout] (default task-10) 	at co.elastic.apm.agent.servlet.ServletTransactionHelper.normalizeServletPath(ServletTransactionHelper.java:217) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,198 INFO  [stdout] (default task-10) 	at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:248) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,198 INFO  [stdout] (default task-10) 	at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,198 INFO  [stdout] (default task-10) 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,199 INFO  [stdout] (default task-10) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
09:36:37,199 INFO  [stdout] (default task-10) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
09:36:37,199 INFO  [stdout] (default task-10) 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
09:36:37,199 INFO  [stdout] (default task-10) 2023-07-05 09:36:37,102 [default task-10] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Advice threw an exception, this should never happen!
09:36:37,200 INFO  [stdout] (default task-10) java.lang.StringIndexOutOfBoundsException: String index out of range: -1
09:36:37,200 INFO  [stdout] (default task-10) 	at java.lang.String.substring(String.java:1967) ~[?:1.8.0_302]
09:36:37,200 INFO  [stdout] (default task-10) 	at co.elastic.apm.agent.servlet.ServletTransactionHelper.normalizeServletPath(ServletTransactionHelper.java:217) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,200 INFO  [stdout] (default task-10) 	at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:248) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,200 INFO  [stdout] (default task-10) 	at co.elastic.apm.agent.servlet.JavaxServletApiAdvice.onExitServletService(JavaxServletApiAdvice.java:45) ~[elastic-apm-agent.jar:1.39.1-SNAPSHOT]
09:36:37,200 INFO  [stdout] (default task-10) 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-9.0.2.Final.jar!/:9.0.2.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) [undertow-servlet-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) [undertow-core-1.2.9.Final.jar!/:1.2.9.Final]
09:36:37,200 INFO  [stdout] (default task-10) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
09:36:37,200 INFO  [stdout] (default task-10) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
09:36:37,200 INFO  [stdout] (default task-10) 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
[INFO ]2023-07-05 09:36:37,198 [default task-13] DEBUG co.elastic.apm.agent.impl.transaction.Span - startSpan '' 00-61cb4b1a1f3a6f4033365ea2cfd77b07-9f8c341c020c8c72-01 (363ec26f)

Sylvain_Juge · July 6, 2023, 3:01pm

Perfect, I did not noticed the extra / at the end of the context path that still triggers the issue:

requestURI = /api-saude-outbox-process/processEvent/40890606
contextPath = /api-saude-outbox-process/ <----- this is where I missed the extra / in context path
servletPath =
pathInfo = /processEvent/40890606

I've updated the tests in and it should be better now.

Could you try to validate with this snapshot and tell me if that fixes it for good now ?

miguel.longo · July 6, 2023, 5:08pm

It's perfect now! I can see all transactions without problems.

I changed the log level to try to find some error, but it's all clean.

Thanks, Sylvain!

Sylvain_Juge · July 7, 2023, 6:51am

Perfect! Thanks again for reporting this, we will likely include the bugfix to the next release.
Make sure to subscribe to notifications on github so you know when to update.

Have a good day!

system · August 4, 2023, 6:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.