No soap web service traces from Jboss Fuse

We've got APM up and running for a few apps, but struggle to get any traces from our Jboss Fuse (6.2.1) servers. APM/Kibana/Elastic v. 7.16.2, most recent build of apm agent (tried older ones too) - 1.28.4.

The applications / webservices run on JBoss Fuse / Apache Karaf in the form of OSGI bundles and apache CXF.

It seems that APM is correctly identifying types/methods:

2022-02-07 12:11:03,411 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.society.ws.impl.SocietyWebService
2022-02-07 12:11:03,427 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.society.ws.impl.SocietyWebService.getSocietiesBySocietyCode()
2022-02-07 12:11:03,458 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.society.ws.impl.SocietyWebService
2022-02-07 12:11:03,458 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.society.ws.impl.SocietyWebService.getSocietiesBySocietyCode()
2022-02-07 12:11:04,802 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.usermanagement.ws.impl.UserManagementWebService
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public my.org.usermanagement.api.domain.UserDTO my.org.usermanagement.ws.impl.UserManagementWebService.findUserById(my.org.usermanagement.ws.dto.FindUserByIdInDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public my.org.usermanagement.api.domain.UserDTO my.org.usermanagement.ws.impl.UserManagementWebService.findUserByLoginName(my.org.usermanagement.ws.dto.FindUserByLoginNameInDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.findUsers(my.org.usermanagement.api.domain.UserSearchCriteriaDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.listUsers(my.org.usermanagement.api.domain.UserListCriteriaDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public my.org.usermanagement.api.domain.OrganizationalUnitDTO my.org.usermanagement.ws.impl.UserManagementWebService.findOrganizationalUnitById(my.org.usermanagement.ws.dto.FindOrganizationalUnitByIdInDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public my.org.usermanagement.api.domain.OrganizationalUnitDTO my.org.usermanagement.ws.impl.UserManagementWebService.findOrganizationalUnitByCode(my.org.usermanagement.ws.dto.FindOrganizationalUnitByCodeInDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.findOrganizationalUnitsByParent(my.org.usermanagement.ws.dto.FindOrganizationalUnitsByParentInDTO)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.listOrganizationalUnits()
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.getRolesForUser(java.lang.Short)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.getTenantsForUser(java.lang.Short)
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.getCurrentRoles()
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.getCurrentTenants()
2022-02-07 12:11:04,817 [Blueprint Extender: 2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JaxWsTransactionNameInstrumentation: (co.elastic.apm.agent.bci.bytebuddy.MethodHierarchyMatcher@6de4f6ad and not(isAbstract())) matches public java.util.List my.org.usermanagement.ws.impl.UserManagementWebService.getAllTenants()
2022-02-07 12:11:05,556 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.agreement.services.impl.ScheduleService
2022-02-07 12:11:05,712 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.work.services.impl.WorkService
2022-02-07 12:11:06,056 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.agreement.services.impl.ScheduleService
2022-02-07 12:11:06,212 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation JaxWsTransactionNameInstrumentation: (declaresAnnotations(whereOne(ofAnnotationType(name(in(javax.jws.WebService, jakarta.jws.WebService))))) and not(isInterface())) matches class my.org.work.services.impl.WorkService

The whole buch of services seems to be listed correctly. There's also something happening with the classloaders:

2022-02-07 14:29:31,360 [CM Configuration Updater (ManagedService Update: pid=[org.apache.cxf.osgi])] DEBUG co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory - Creating plugin class loader for org.apache.cxf.cxf-rt-transports-http [96] containing [co.elastic.apm.agent.servlet.AsyncInstrumentation$AsyncContextInstrumentation, co.elastic.apm.agent.servlet.JakartaFilterInstrumentation, co.elastic.apm.agent.servlet.FilterChainInstrumentation, co.elastic.apm.agent.servlet.ServletVersionInstrumentation, co.elastic.apm.agent.servlet.JakartaRequestStreamRecordingInstrumentation, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation$JavaxStartAsyncInstrumentation$StartAsyncAdvice, co.elastic.apm.agent.servlet.RequestStreamRecordingInstrumentation, co.elastic.apm.agent.servlet.JakartaFilterChainInstrumentation, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaInit, co.elastic.apm.agent.servlet.AbstractServletInstrumentation, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxInit, co.elastic.apm.agent.servlet.Constants, co.elastic.apm.agent.servlet.JavaxFilterChainInstrumentation, co.elastic.apm.agent.servlet.JakartaRequestStreamRecordingInstrumentation$GetInputStreamAdvice, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation, co.elastic.apm.agent.servlet.RequestDispatcherSpanType, co.elastic.apm.agent.servlet.ServletApiAdvice, co.elastic.apm.agent.servlet.ServletVersionInstrumentation$Init, co.elastic.apm.agent.servlet.JavaxServletApiAdvice, co.elastic.apm.agent.servlet.JavaxRequestStreamRecordingInstrumentation$GetInputStreamAdvice, co.elastic.apm.agent.servlet.FilterInstrumentation, co.elastic.apm.agent.servlet.package-info, co.elastic.apm.agent.servlet.ServletInstrumentation, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaInit$AdviceClass, co.elastic.apm.agent.servlet.JakartaServletApiAdvice, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaService, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation$JakartaAsyncContextInstrumentation, co.elastic.apm.agent.servlet.JavaxFilterInstrumentation, co.elastic.apm.agent.servlet.JakartaUtil, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation, co.elastic.apm.agent.servlet.AsyncInstrumentation$StartAsyncInstrumentation, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation$JavaxStartAsyncInstrumentation, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation, co.elastic.apm.agent.servlet.RequestStreamRecordingInstrumentation$GetInputStreamAdvice, co.elastic.apm.agent.servlet.ServletTransactionHelper, co.elastic.apm.agent.servlet.AsyncInstrumentation, co.elastic.apm.agent.servlet.JakartaServletInstrumentation, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation$JakartaStartAsyncAdvice, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation$JavaxAsyncContextInstrumentation, co.elastic.apm.agent.servlet.JavaxRequestStreamRecordingInstrumentation, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaService$AdviceClass, co.elastic.apm.agent.servlet.AsyncInstrumentation$AsyncContextInstrumentation$AsyncContextStartAdvice, co.elastic.apm.agent.servlet.ServletHelper, co.elastic.apm.agent.servlet.ServletVersionInstrumentation$Service, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation, co.elastic.apm.agent.servlet.helper.JavaxServletRequestHeaderGetter, co.elastic.apm.agent.servlet.helper.JakartaRecordingServletInputStreamWrapper, co.elastic.apm.agent.servlet.helper.JakartaServletRequestHeaderGetter, co.elastic.apm.agent.servlet.helper.CommonServletRequestHeaderGetter, co.elastic.apm.agent.servlet.helper.JakartaServletTransactionCreationHelper, co.elastic.apm.agent.servlet.helper.JakartaApmAsyncListener, co.elastic.apm.agent.servlet.helper.ServletTransactionCreationHelper, co.elastic.apm.agent.servlet.helper.package-info, co.elastic.apm.agent.servlet.helper.JakartaAsyncContextAdviceHelper$1, co.elastic.apm.agent.servlet.helper.JavaxApmAsyncListener, co.elastic.apm.agent.servlet.helper.JavaxAsyncContextAdviceHelper$1, co.elastic.apm.agent.servlet.helper.AsyncContextAdviceHelper, co.elastic.apm.agent.servlet.helper.AsyncConstants, co.elastic.apm.agent.servlet.helper.JavaxAsyncContextAdviceHelper, co.elastic.apm.agent.servlet.helper.JavaxAsyncContextAdviceHelper$ApmAsyncListenerAllocator, co.elastic.apm.agent.servlet.helper.JakartaAsyncContextAdviceHelper$JakartaApmAsyncListenerAllocator, co.elastic.apm.agent.servlet.helper.JavaxServletTransactionCreationHelper, co.elastic.apm.agent.servlet.helper.JavaxRecordingServletInputStreamWrapper, co.elastic.apm.agent.servlet.helper.JakartaAsyncContextAdviceHelper, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxService, co.elastic.apm.agent.servlet.JavaxServletInstrumentation, co.elastic.apm.agent.servlet.JavaxUtil, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxService$AdviceClass, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxInit$AdviceClass, co.elastic.apm.agent.bci.classloading.LookupExposer]
2022-02-07 14:29:31,423 [CM Configuration Updater (ManagedService Update: pid=[org.apache.cxf.osgi])] DEBUG co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory - Creating plugin class loader for org.apache.geronimo.specs.geronimo-servlet_3.0_spec [82] containing [co.elastic.apm.agent.servlet.AsyncInstrumentation$AsyncContextInstrumentation, co.elastic.apm.agent.servlet.JakartaFilterInstrumentation, co.elastic.apm.agent.servlet.FilterChainInstrumentation, co.elastic.apm.agent.servlet.ServletVersionInstrumentation, co.elastic.apm.agent.servlet.JakartaRequestStreamRecordingInstrumentation, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation$JavaxStartAsyncInstrumentation$StartAsyncAdvice, co.elastic.apm.agent.servlet.RequestStreamRecordingInstrumentation, co.elastic.apm.agent.servlet.JakartaFilterChainInstrumentation, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaInit, co.elastic.apm.agent.servlet.AbstractServletInstrumentation, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxInit, co.elastic.apm.agent.servlet.Constants, co.elastic.apm.agent.servlet.JavaxFilterChainInstrumentation, co.elastic.apm.agent.servlet.JakartaRequestStreamRecordingInstrumentation$GetInputStreamAdvice, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation, co.elastic.apm.agent.servlet.RequestDispatcherSpanType, co.elastic.apm.agent.servlet.ServletApiAdvice, co.elastic.apm.agent.servlet.ServletVersionInstrumentation$Init, co.elastic.apm.agent.servlet.JavaxServletApiAdvice, co.elastic.apm.agent.servlet.JavaxRequestStreamRecordingInstrumentation$GetInputStreamAdvice, co.elastic.apm.agent.servlet.FilterInstrumentation, co.elastic.apm.agent.servlet.package-info, co.elastic.apm.agent.servlet.ServletInstrumentation, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaInit$AdviceClass, co.elastic.apm.agent.servlet.JakartaServletApiAdvice, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaService, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation$JakartaAsyncContextInstrumentation, co.elastic.apm.agent.servlet.JavaxFilterInstrumentation, co.elastic.apm.agent.servlet.JakartaUtil, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation, co.elastic.apm.agent.servlet.AsyncInstrumentation$StartAsyncInstrumentation, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation$JavaxStartAsyncInstrumentation, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation, co.elastic.apm.agent.servlet.RequestStreamRecordingInstrumentation$GetInputStreamAdvice, co.elastic.apm.agent.servlet.ServletTransactionHelper, co.elastic.apm.agent.servlet.AsyncInstrumentation, co.elastic.apm.agent.servlet.JakartaServletInstrumentation, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation$JakartaStartAsyncAdvice, co.elastic.apm.agent.servlet.JavaxAsyncInstrumentation$JavaxAsyncContextInstrumentation, co.elastic.apm.agent.servlet.JavaxRequestStreamRecordingInstrumentation, co.elastic.apm.agent.servlet.JakartaServletVersionInstrumentation$JakartaService$AdviceClass, co.elastic.apm.agent.servlet.AsyncInstrumentation$AsyncContextInstrumentation$AsyncContextStartAdvice, co.elastic.apm.agent.servlet.ServletHelper, co.elastic.apm.agent.servlet.ServletVersionInstrumentation$Service, co.elastic.apm.agent.servlet.JakartaAsyncInstrumentation, co.elastic.apm.agent.servlet.helper.JavaxServletRequestHeaderGetter, co.elastic.apm.agent.servlet.helper.JakartaRecordingServletInputStreamWrapper, co.elastic.apm.agent.servlet.helper.JakartaServletRequestHeaderGetter, co.elastic.apm.agent.servlet.helper.CommonServletRequestHeaderGetter, co.elastic.apm.agent.servlet.helper.JakartaServletTransactionCreationHelper, co.elastic.apm.agent.servlet.helper.JakartaApmAsyncListener, co.elastic.apm.agent.servlet.helper.ServletTransactionCreationHelper, co.elastic.apm.agent.servlet.helper.package-info, co.elastic.apm.agent.servlet.helper.JakartaAsyncContextAdviceHelper$1, co.elastic.apm.agent.servlet.helper.JavaxApmAsyncListener, co.elastic.apm.agent.servlet.helper.JavaxAsyncContextAdviceHelper$1, co.elastic.apm.agent.servlet.helper.AsyncContextAdviceHelper, co.elastic.apm.agent.servlet.helper.AsyncConstants, co.elastic.apm.agent.servlet.helper.JavaxAsyncContextAdviceHelper, co.elastic.apm.agent.servlet.helper.JavaxAsyncContextAdviceHelper$ApmAsyncListenerAllocator, co.elastic.apm.agent.servlet.helper.JakartaAsyncContextAdviceHelper$JakartaApmAsyncListenerAllocator, co.elastic.apm.agent.servlet.helper.JavaxServletTransactionCreationHelper, co.elastic.apm.agent.servlet.helper.JavaxRecordingServletInputStreamWrapper, co.elastic.apm.agent.servlet.helper.JakartaAsyncContextAdviceHelper, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxService, co.elastic.apm.agent.servlet.JavaxServletInstrumentation, co.elastic.apm.agent.servlet.JavaxUtil, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxService$AdviceClass, co.elastic.apm.agent.servlet.JavaxServletVersionInstrumentation$JavaxInit$AdviceClass, co.elastic.apm.agent.bci.classloading.LookupExposer]
2022-02-07 14:30:05,677 [qtp1272417379-166] DEBUG co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory - Creating plugin class loader for phx-businessintelligence-ws [431] containing [co.elastic.apm.agent.jaxws.package-info, co.elastic.apm.agent.jaxws.JaxWsTransactionNameInstrumentation$AdviceClass, co.elastic.apm.agent.jaxws.JaxWsTransactionNameInstrumentation, co.elastic.apm.agent.bci.classloading.LookupExposer]
2022-02-07 14:30:27,409 [qtp1272417379-119] DEBUG co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory - Creating plugin class loader for phx-businesscase-ws [417] containing [co.elastic.apm.agent.jaxws.package-info, co.elastic.apm.agent.jaxws.JaxWsTransactionNameInstrumentation$AdviceClass, co.elastic.apm.agent.jaxws.JaxWsTransactionNameInstrumentation, co.elastic.apm.agent.bci.classloading.LookupExposer]

The whole bunch of JMS based camel routes running on the same instance that are being traced correctly. But I can't see any transaction/span related messages in regard to jax-ws whatsoever.

There's a couple of these:

2022-02-07 14:29:45,329 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation JmsMessageListenerInstrumentation: (name(equals(onMessage)) and hasParameter(hasTypes(with(0 matches erasure(hasSuperType(erasure(name(equals(javax.jms.Message)))))))) and isPublic() and not(isAbstract())) matches public void org.apache.camel.component.jms.EndpointMessageListener.onMessage(javax.jms.Message)
2022-02-07 14:29:45,439 [FelixStartLevel] INFO  co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: FakeURLStreamHandler can not be used!
2022-02-07 14:29:45,439 [FelixStartLevel] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Implementation version parsing error: [org.apache.httpcomponents.httpclient [289]]
java.io.IOException: FakeURLStreamHandler can not be used!
	at org.apache.felix.framework.FakeURLStreamHandler.openConnection(FakeURLStreamHandler.java:39) ~[?:?]
	at java.net.URL.openConnection(Unknown Source) ~[?:1.8.0_91]
	at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers.readImplementationVersionFromManifest(CustomElementMatchers.java:197) ~[elastic-apm-agent-1.28.4.jar:?]
	at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers.access$100(CustomElementMatchers.java:51) ~[elastic-apm-agent-1.28.4.jar:?]
	at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers$3.matches(CustomElementMatchers.java:150) [elastic-apm-agent-1.28.4.jar:?]

But aside from that and reoccuring "cannot load JBoss Logging/log4j/log4j2 API" errors, there's not much to work with...

Any suggestions? I'm out of ideas and specially this old legacy stuff is where APM would really come in handy..

We don't have dedicated support for JBoss Fuse and unfortunately there is no intention to add one.

However, you may use the camel OpenTracing component and our OpenTracing integration. See some explanation on that on a related topic.

Alternatively, you may be able to add a custom interceptor, in which you can use our public API to create and manage (activate, close, end) transactions.

Once you have something that creates and activates transactions, all other supported technologies will just work as well (for example - JDBC or HTTP clients).

I hope this helps

So as long as a transaction is created and closed everything in between will be captured? I.e. JDBC/JPA calls, method calls and such?

Everything that is listed in the supported technologies.

Thanks! Using an interceptor was a great idea! We already have "in" and "out" interceptors for logging purposes, so we can integrate apm into those easily without having to touch much code.

Bonus question though: is there any way to make every method call within my created transaction a span?

It's nice to see the database instrumentation and all, but the info feels worthless without knowding the business logic flow involved. We'd need to get the surrounding methods. What I'd like to avoid is having to annotate every single business method (touching several hundred classes) with @Span or such...

That's awesome!

There is, but that would be a terrible idea both in terms of performance and usability. You'd be surprised how many method calls there are even within a very basic code path.

Please take a look at this method-tracing guide. Your best bet would be the sampling profiler. In order to always get specific methods, either use the public API or the trace_methods config, but if you use the latter, please take extra care to be very specific and not capture entire packages.

I hope this helps.

Yeah, that's kind of my problem. Sampling profiler doesn't work, since the application server runs on Windows.
Tracing configuration wouldn't work either, because it's nearly impossible to hit just the right amount of methods.

My problem is a different one: when using both, the API to manually start a transaction and trace_methods config, it seems that the manually created transaction is lost, overriden by trace_methods. And a bunch of other transactions are being captured alongside. What I'd like to have is only my custom transaction being captured/displayed as such and no instrumentation/sampling of methods (matching the trace_methods) taking place, unless a manually created transaction exists. This would narrow it down to quite a few method calls.

TL;DR - don't create transactions automatically, only create spans (according to trace_methods) within a pre-existing transaction. Is that possible?

don't create transactions automatically, only create spans (according to trace_methods) within a pre-existing transaction. Is that possible?

No, that's not possible. The best option for you then is to find better values for trace_methods that are more specific to the business methods within transactions.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.