Monitoring JBoss Fuse using APM

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

Kibana version: 6.5.4

Elasticsearch version: 6.5.4

APM Server version: 6.8.0

APM Agent language and version: 1.7.0, Java. (tried apm-agent-attach-1.6.1.jar also)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum

Fresh install or upgraded from other version? fresh

Hi All,

Firstly, I want to ask the below question

Can Elastic APM be used to monitor Red Hat JBoss Fuse product (which has cxf endpoints for web services, Active MQ for messaging, ESB, etc)?

I installed APM server and tried the below 2 scenarios to monitor the above use case

Scenario 1: I attached the agent to a running PID of a container in Fuse which has OSGi bundle deployed in it. These bundles include cxf web services, etc

java -jar apm-agent-attach-1.6.1.jar --pid 1256 -Delastic.apm.service_name=fuse-service -Delastic.apm.server_urls=http://192.168.0.20:8200'

The agent was attached successfully but I could not see any services in Kibana APM. Get No services were found

Scenario 2: I use it as java agent in jvm options

-javaagent:"D:\APM-Agent\elastic-apm-agent-1.7.0.jar"

The container in Fuse did not start for this scenario.

Can anyone help or guide how to use Elastic APM in monitoring RedHat Fuse ?

Thanks,

Hi :slight_smile:

We currently don't offer dedicated support for Red Hat Fuse.

This is not how to define parameters for attachment. Use the --args or --config options: https://www.elastic.co/guide/en/apm/agent/java/current/setup-attach-cli.html

Could you post your logs?

Thanks,
Felix

Thanks for your response. @felixbarny

Sorry, I did run the command with --args option, but it prompted me to use --config instead.

After running with --config, the result was same. Get No services were found message.

For 2nd scenario, snippet from the fuse container logs ...

2019-06-17 14:30:22,331 | ERROR | FelixStartLevel | BlueprintContainerImpl | 17 - org.apache.aries.blueprint.core - 1.4.5 | Unable to start blueprint container for bundle agrobank-ws-proxy/3.1.0.SNAPSHOT
org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean .camelBlueprint.factory.ws-proxy-context
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:738)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:848)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)[17:org.apache.aries.blueprint.core:1.4.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_121]
at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:247)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintRepository.createInstance(BlueprintRepository.java:230)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintRepository.create(BlueprintRepository.java:145)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.getComponentInstance(BlueprintContainerImpl.java:760)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.camel.blueprint.handler.CamelNamespaceHandler$CamelDependenciesFinder.process(CamelNamespaceHandler.java:965)[132:org.apache.camel.camel-blueprint:2.15.1.redhat-621222]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.processProcessors(BlueprintContainerImpl.java:535)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:367)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:270)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:294)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:263)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:253)[17:org.apache.aries.blueprint.core:1.4.5]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)[11:org.apache.aries.util:1.1.3]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)[11:org.apache.aries.util:1.1.3]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[11:org.apache.aries.util:1.1.3]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[11:org.apache.aries.util:1.1.3]
at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)[11:org.apache.aries.util:1.1.3]
at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1127)[org.apache.felix.framework-4.4.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:696)[org.apache.felix.framework-4.4.1.jar:]
at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:484)[org.apache.felix.framework-4.4.1.jar:]
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4429)[org.apache.felix.framework-4.4.1.jar:]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2100)[org.apache.felix.framework-4.4.1.jar:]
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1299)[org.apache.felix.framework-4.4.1.jar:]
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)[org.apache.felix.framework-4.4.1.jar:]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentHashMapV8
at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.(ConcurrentLinkedHashMap.java:221)
at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.(ConcurrentLinkedHashMap.java:104)
at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Builder.build(ConcurrentLinkedHashMap.java:1598)
at org.apache.camel.util.LRUCache.(LRUCache.java:84)
at org.apache.camel.impl.DefaultEndpointRegistry.(DefaultEndpointRegistry.java:44)
at org.apache.camel.impl.DefaultCamelContext.(DefaultCamelContext.java:267)
at org.apache.camel.blueprint.BlueprintCamelContext.(BlueprintCamelContext.java:62)
at org.apache.camel.blueprint.CamelContextFactoryBean.createContext(CamelContextFactoryBean.java:209)
at org.apache.camel.blueprint.CamelContextFactoryBean.getContext(CamelContextFactoryBean.java:192)
at org.apache.camel.blueprint.CamelContextFactoryBean.getContext(CamelContextFactoryBean.java:76)
at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.getContext(AbstractCamelContextFactoryBean.java:599)
at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.initLazyLoadTypeConverters(AbstractCamelContextFactoryBean.java:383)
at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.afterPropertiesSet(AbstractCamelContextFactoryBean.java:164)
at org.apache.camel.blueprint.CamelContextFactoryBean.afterPropertiesSet(CamelContextFactoryBean.java:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_121]
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:736)
... 29 more

Hope you can help.

Thanks,

Please upload the full log.

@Eyal_Koren
How do I upload full log file containing approx 200000 characters?

Currently its allows only 7000 characters.

Maybe through gist?
Otherwise, add as much as you can from the part you already did and below

Thanks @Eyal_Koren

Here is the gist link

Looks like an OSGi issue.
Please set the boot_delegation_packages config property with the value suggested there to use for Pentaho and let me know what happens.

Hi,

I set the boot_delegation_packages property as per your suggestion. The fuse container did not start. Had same errors as before.

Here is the gist link:

This is how I configure java agent for the fuse container under jvmSettings.

FYI, the value entered in the jvmSettings field is

-javaagent:D:\Beans\APMAgent\elastic-apm-agent-1.7.0.jar -Delastic.apm.service_name=fuse-ws-service -Delastic.apm.application_packages=org.example,org.another.example -Delastic.apm.server_urls=http://192.168.0.38:8200 -Delastic.apm.log_level=error -Delastic.apm.boot_delegation_packages=org.apache.karaf.jaas.boot,org.apache.karaf.jaas.boot.principal,org.apache.karaf.management.boot,sun.,com.sun.,javax.transaction,javax.transaction.,javax.xml.crypto,javax.xml.crypto.,org.apache.xerces.jaxp.datatype,org.apache.xerces.stax,org.apache.xerces.parsers,org.apache.xerces.jaxp,org.apache.xerces.jaxp.validation,org.apache.xerces.dom,co.elastic.apm.agent.*

Thank you

The bootdelegation value in the documentation page unfortunately doesn't render properly. The correct value would be this one:

org.apache.karaf.jaas.boot,org.apache.karaf.jaas.boot.principal,org.apache.karaf.management.boot,sun.*,com.sun.*,javax.transaction,javax.transaction.*,javax.xml.crypto,javax.xml.crypto.*,org.apache.xerces.jaxp.datatype,org.apache.xerces.stax,org.apache.xerces.parsers,org.apache.xerces.jaxp,org.apache.xerces.jaxp.validation,org.apache.xerces.dom,co.elastic.apm.agent.*

could you retry with that?

Yes. I realized it later.

Thanks, it works with the given value. I applied the javaagent to web service and esb containers. Containers did start up and were running. However I have some questions

  • I did not see any APM agent related logging in the container logs. (e.g. starting apm agent or xyz metrics being monitored etc), Why is that?
  • Although the APM Server Status and APM Agent checks are OK, I don't see anything in KIbana's APM menu. What am I supposed to see here?

The graphs don't change if I fire sample requests to the web service.

  • How do I configure custom metrics in APM and then visualize in Kibana? Is it available in Basic version?
  • Can apm javaagent monitor metrics like ExchangesCompleted, MeanProcessingTime, ExchangesTotal, QueueSize etc in a camel context or camel route? Just the way it can be done in Grafana using Prometheus engine.

container log

Thanks for your time

You set -Delastic.apm.log_level=error so you would only get logging about errors, and apparently there aren't such

Did you go over our supported-technologies page? What are you using from there? Please go over it and see what is expected to be traced and is not. And if not- please add a log where log_level is set to DEBUG.
Since you are using RedHat JBoss Fuse, you are probably not using Servlets, and JMS support is off by default in 1.7.0, as it is still experimental, so there is no way for automatic creation of transactions. If you are using JMS, please see in the documentation what is supported and how to turn this on.
If you are using JBoss Fuse Service Works, which relies on JBoss EAP as its container, you may get automatic traces.
Otherwise, go over our public API to see how you can manually trace your code.

As replied at the beginning of this thread- "We currently don't offer dedicated support for Red Hat Fuse".

We don't have a way to configure custom metrics as of yet.

See a list of supported metrics in our documentation.

I hope this helps.

I tried all log levels but the result is same. The container starts as before but no apm related logging found in log file.

However FYI, instead of applying javaagent and other Elastic APM configurations to a child container, I tried to apply the settings to a root container (Karaf) and I can see elastic apm related logging in the root container's log file. (gist url for this is given below)

But in reality, one should not be applying apm javaagent settings to a root container in Fuse.

It should be applied to required child containers where we want to gather relevant metrics from. (e.g. cxf endpoints etc)

gist url :

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.