Support for automatic JMS monitoring in APM 6.7.1

Elastic Observability APM
1

Hi,

We are evaluating ELK stack v6.7.1 (this is our companies current production version).
We have an application which uses JMS (topics and queues). But I dont see any transactions of type "messaging". Does v6.7.1. support this? Do I have to use trace_methods config param?

So far I've used the following trace_methods options without success:

public com.example.app.package1.AJmsListener#onMessage, public @javax.ejb.MessageDriven com.example.app., public javax.jms.MessageListener#onMessage com.example.app.

Transactions of type "request" and some other "custom" are being captured but no "messaging".

Should the transactions appear in the Kibana APM dashboard in the "filter by type" dropdown as type "messaging"?

My complete elasticapm.properties:

service_name=myservice
service_version=1.0.0
application_packages=com.example.app
server_urls=http://aserver.local:8200
trace_methods=public com.example.app.package1.AnotherJmsListener#onMessage, public @javax.ejb.MessageDriven com.example.app., public javax.jms.MessageListener#onMessage com.example.app.

Best regards,
Murat

2

Welcome to our forum Murat!
JMS is on by default only since the Java agent version 1.13.0, so make sure you are using this one.
The JMS tracing should be done automatically, so you can (and should) remove these trace_methods settings.
If you are using version 1.13.0 and it doesn't work, please set log_level to DEBUG and upload the logs.
I hope this helps.
Eyal.

3

Okay. I was using 1.11.0 because I have had problems with 1.13.0.
Is agent 1.13.0 compatible with APM server 6.7.1?

If I install v.1.13.0 of the agent via the application deployment in Weblogic Server 12.2.1.3.0 (maven dependency in the java project) most of the time, the deployment of the app will fail. After restarting the deployment, deployment will succeed but this is not always the case. No traces appear in APM Server

The errors I get in Weblogic 12.2.1.3.0:

  • thread dumps
  • BEA: ServerIdentity failed validation, downgrading to anonymous.

Agent version 1.11.0 does work most of the time. Sometimes I have to restart/redeploy the application. Also here I often see a threaddump.

This is the apm-server log:

2020-02-27T09:56:58.881Z WARN elasticsearch/client.go:539 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x2e40d200, ext:63718394215, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"span":common.MapStr{"self_time":common.MapStr{"count":1, "sum":common.MapStr{"us":1417}}}, "context":common.MapStr{"process":common.MapStr{"pid":26788, "title":"/xx/xx/oracle/jdk/jre/bin/java"}, "service":common.MapStr{"runtime":common.MapStr{"name":"Java", "version":"1.8.0_231"}, "name":"xxxxxxx", "agent":common.MapStr{"name":"java", "version":"1.13.0"}, "version":"1.0.0", "language":common.MapStr{"name":"Java", "version":"1.8.0_231"}}, "system":common.MapStr{"platform":"Linux", "ip":"xxxxxx", "hostname":"xxxxxx", "architecture":"amd64"}}, "processor":common.MapStr{"name":"metric", "event":"metric"}, "beat":common.MapStr{"name":"xxxx", "hostname":"xxxxx", "version":"6.7.1"}, "host":common.MapStr{"name":"xxxxx"}}, Private:interface {}(nil)}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

4

Well, this doesn't sound right...
What do the thread dumps say?
Does your application deployment only fails when an agent is installed?
Is the Java version you are using 1.8.0_231?

In order for us to be able to say something, please:

  • Remove the current trace_methods setting
  • Set log_level=DEBUG
  • Reproduce deployment failure and send the application+agent logs as well as the thread dump
5

Hi Eyal,

What do the thread dumps say?

  • lots of threads waiting on other trx. No "BLOCKED" status although; (i'm not an expert on reading a thread dump, sorry)
    Does your application deployment only fails when an agent is installed?
  • yes it seems; deployment is usually okay. On server restart deployments won't start sometimes (1 out of 3-4-5 times?)
    Is the Java version you are using 1.8.0_231 ?
  • yes

I have to ask for permissions to post logs on this forum. I will let you know.

Is the Programmatic API setup method mentioned here still experimental?
We do not see any java.lang.NoClassDefFoundError within Weblogic AS v12.2.
What we see is often a Thread dump and this error message after deployment or after restart of the server:

Failed to initialize the application "wsm-pm" due to error weblogic.application.ModuleException: Error: Unresolved Webapp Library references for "WebAppModule(wsm-pm:/idaas/webservice)", defined in weblogic.xml [Extension-Name: oracle.wsm.idmrest.sharedlib, Specification-Version: 1, Implementation-Version: 12.2.1.3, exact-match: false]

6

Sorry, I missed that before. You should not attach the agent programatically through a webapp deployment.
For WebLogic, it's best if you install the agent using the -javaagent option.
If this option is invalid for you, you can also remotely attach the agent.

I hope this helps.

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.