Java APM attached agent not launching

Elastic Observability APM
1

Hello there,

I am trying to use the Automatic setup with apm-agent-attach-standalone.jar. I am running the agent with Oracle Java 8, and the java application I want to monitor is running with Java 7.

Kibana version: 7.9.0

Elasticsearch version: 7.9.2

APM Server version: 7.9.2

APM Agent language and version: Java Oracle 8, 1.18.0

Browser version: Google Chrome 80.0.3987.149

Original install method (e.g. download page, yum, deb, from source, etc.) and version: The agent was installed from the link: https://search.maven.org/remotecontent?filepath=co/elastic/apm/apm-agent-attach/1.18.0/apm-agent-attach-1.18.0-standalone.jar

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? The communication between the agent and the APM server is plain within an internal network. The problem is the agent won't attach to the Java VM, so it has nothing to do with a communication issue with APM server.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

[root@company jboss-as-7.1.1]# java -jar apm-agent-attach-1.18.0-standalone.jar --pid 27769 --config service_name=comerzzia-pre server_urls=http://192.168.250.114:8200
2020-10-01 13:16:17.631  INFO Attaching the Elastic APM agent to 27769
Exception in thread "main" java.lang.IllegalStateException: Error during attachment using: co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@3f8f9dd6
        at co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:613)
        at co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:273)
        at co.elastic.apm.attach.ElasticApmAttacher.attach(ElasticApmAttacher.java:159)
        at co.elastic.apm.attach.RemoteAttacher.main(RemoteAttacher.java:88)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at co.elastic.apm.attach.bytebuddy.agent.Attacher.install(Attacher.java:106)
        at co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:608)
        ... 3 more
Caused by: java.lang.IllegalStateException: Target VM did not respond: 27769
        at co.elastic.apm.attach.bytebuddy.agent.VirtualMachine$ForHotSpot$Connection$Factory$ForSocketFile.connect(VirtualMachine.java:541)
        at co.elastic.apm.attach.bytebuddy.agent.VirtualMachine$ForHotSpot.attach(VirtualMachine.java:256)
        at co.elastic.apm.attach.bytebuddy.agent.VirtualMachine$ForHotSpot.attach(VirtualMachine.java:243)
        ... 9 more

Apparently, no matter the java version I use when launching the agent (Java 7 or Java 8), it won't attach to the JVM of the Java 7 process.
The command I use to launch the agent is the following:
java -jar $(pwd)/apm-agent-attach-1.18.0-standalone.jar --pid $(ps -C java -f | grep -i standalone | awk '{print $2}') --config service_name=comerzzia-pre server_urls=http://192.168.250.114 :8200 applicaion_packages=bin.comerzzia,org.apache.jsp.backoffice,org.jboss.byteman log_file=/apps/bin/comerzzia/jboss-as-7.1.1/standalone/log/apm.log log_level=DEBUG

Is there any additional step I am missing? Any help provided is welcome!

Thanks in advance!

2

Hi and welcome to our forum!

Try running the standalone jar with the same Java you use to run the application

3

Hello Eyal, thanks for your quick response!

Apparenty, the error is still the same.

[root@company jboss-as-7.1.1]# ps -C java -o pid,cmd
  PID CMD
27769 /apps/bin/comerzzia/jdk1.7.0_80/bin/java -D[Standalone] -server -XX:+UseCompressedOops -XX:+TieredCompilation -Duser.timezone=Europe/Madrid -Xms8192m -Xmx8192m -XX:MaxPermSize=2048m -
[root@company jboss-as-7.1.1]# /apps/bin/comerzzia/jdk1.7.0_80/bin/java -jar apm-agent-attach-1.18.0-standalone.jar --pid 27769 --config service_name=comerzzia-pre server_urls=http://192.168.250.114:8200
2020-10-01 13:54:19.797  INFO Attaching the Elastic APM agent to 27769
Exception in thread "main" java.lang.IllegalStateException: Error during attachment using: co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@3d6998b9
        at co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:613)
        at co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:273)
        at co.elastic.apm.attach.ElasticApmAttacher.attach(ElasticApmAttacher.java:159)
        at co.elastic.apm.attach.RemoteAttacher.main(RemoteAttacher.java:88)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at co.elastic.apm.attach.bytebuddy.agent.Attacher.install(Attacher.java:106)
        at co.elastic.apm.attach.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:608)
        ... 3 more
Caused by: java.lang.IllegalStateException: Target VM did not respond: 27769
        at co.elastic.apm.attach.bytebuddy.agent.VirtualMachine$ForHotSpot$Connection$Factory$ForSocketFile.connect(VirtualMachine.java:541)
        at co.elastic.apm.attach.bytebuddy.agent.VirtualMachine$ForHotSpot.attach(VirtualMachine.java:256)
        at co.elastic.apm.attach.bytebuddy.agent.VirtualMachine$ForHotSpot.attach(VirtualMachine.java:243)
        ... 9 more
[root@company jboss-as-7.1.1]#
4

Just saw your command line - seems like you trying to connect to the JVM running the standalone jar...

The process you provide for --pid should be that of the JVM you want to attach to.

Also, you should use the same user to run the standalone jar as the user running the target JVM.

I hope this helps

5

Hello Eyal,

I updated my startup script for the agent so it is runned by jboss user. Btw, jboss is the user that runs the java 7 process I want to monitor. I am still using java 8 to monitor my java 7 server.

The script looks like this:

#!/usr/bin/bash
sudo -E -u jboss  java -jar $(pwd)/apm-agent-attach-1.18.0-standalone.jar --pid $(ps -C java -f | grep -i standalone | awk '{print $2}') --config service_name=comerzzia-pre server_urls=http
://192.168.250.114:8200 applicaion_packages=bin.comerzzia,org.apache.jsp.backoffice,org.jboss.byteman log_file=/apps/bin/comerzzia/jboss-as-7.1.1/standalone/log/apm.log log_level=DEBUG

At this very moment, the aforementioned error of the agent not being able to bind itself with the JVM process has dissappeared. The logs look like the following:

[root@company jboss-as-7.1.1]# ./launch_apm_java_agent.sh 
2020-10-01 16:16:34.367  INFO Attaching the Elastic APM agent to 27769
2020-10-01 16:16:34.635  INFO Done
[root@company jboss-as-7.1.1]#

Is this behaviour normal? Should not the agent be in background monitoring continuously? Besides, I cannot see any log in the path /apps/bin/comerzzia/jboss-as-7.1.1/standalone/log/apm.log

Thanks for your help and support Eyal,

6

Ahh, you grep for the JBoss standalone...

You would need a --config (or -C) before each config option. After you fix that, you should be good to go. Good luck!

7

BTW, since you already attached the agent before, you may need to restart your JBoss this time. There is no detach option. The behaviour you see is normal - the agent is attached and the attaching process terminates. The continuous mode is for cases where you want any new JVM starting up on the host to be attached soon after it starts. When you want to attach to a single process which you know is already running, what you did is the way to go.

8

Hello, I added the --config flag after each of the options.

I assume that the behaviour is normal and it is working as expected.

Summarizing, the idea is that the java process finishes because its main thread only handles the attachment JVM process. Taking this information into account, if I ever want to finish the agent the only thing I have to do is reboot my Java JBOSS applicattion. Am I right? Is there anything I misunderstood?

Thanks for all your help!

Best,

9

That is correct, as long as you don't use it in the continuous mode.

Well, I said there is no complete detach, meaning no way to completely unload agent code from the JVM. However, the is certainly a way to disable the agent in runtime through the recording config. Furthermore, you can reverse agent bytecode instrumentation in runtime through the instrument config.
These config options are dynamic, check out how such options can be utilized.

Good luck!

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.