Elastic APM- apm-agent-attach-cli.jar: Stop collecting metrics through apm cli

[vamshigottam]

We are capturing the application metrics with elastic APM agent using apm-agent-attach-cli.jar which attaches to the running JVM's without making any changes to the code or JVM startup.
This is working fine without any issues and metrics are visible on UI.
however we cannot detach apm-agent-attach-cli.jar from running JVM unless we restart the JVM Process.
Could you share how this can be tackled through apm-agent-attach-cli.jar.

Documentation followed: Automatic setup with apm-agent-attach-cli.jar | APM Java Agent Reference [1.x] | Elastic

command used to attach running JVM

java -jar apm-agent-attach-cli.jar \
    --exclude-user root \
    --include-vmarg XXXXXXXXXX \
    --continuous \
    --config application_packages=XXX \
    --config server_url=http://localhost:8282 \
    --agent-jar elastic-apm-agent-1.43.0.jar &

Kibana version: 8.8.2

Elasticsearch version: 8.8.2

APM Server version: 8.8.2

APM Agent language and version: Java

[Jonas_Kunz]

You can't "detach" the apm-agent from a process. What you can do is the following:

• When attaching, configure the agent to fetch its configuration from a file
• Configure recording to true / false in that file. The agent will periodically reload the file and disable/enable itself accordingly.

[Sandeep_Rudra]

Hi Jonas, Getting the configurations from properties file, APM agent is still pushing metrics to APM server. Is there anything else we need to look at to stop the metrics from being sent to APM server? We also tried disable_send to true and Instrument to false

Properties file:

elastic.apm.server_url=https://dc1-elsticuat08.ulinedm.com:8282
elastic.apm.recording=false
elastic.apm.application_packages=com.uline

Script we are running:

java -jar apm-agent-attach-cli.jar \
    --exclude-user root \
    --include-vmarg consul.port=8500 \
    --continuous \
    --config config_file=true
    --agent-jar elastic-apm-agent.jar

[Jonas_Kunz]

Two things I can spot here:

• config_file is not a boolean option, but a string. It should point to the file location.
• the configuration options in the properties file must be given without the elastic.apm. prefix.

I suggest you fix those two issues and then check the APM Agent debug logs to see whether the options are correctly picked up.

[Sandeep_Rudra]

Thank you Jonas for the reply. I made the changes you suggested, below you should see the poperties and script we used. Still getting trace metrics to APM server.

Output after the script ran:

The tracer said paused but we are still seeing the transactions coming into Kibana APM. Since the agent is already attached it is throwing an Error. but eventually when I left the process running for 1 hour it eventually said tracer paused. Highled in the screenshot below.

Script:

java -jar apm-agent-attach-cli.jar \
    --exclude-user root \
    --include-vmarg consul.port=xxxx\
    --continuous \
    --config config_file="/apm-agent-attach/elasticapm.properties" \
    --agent-jar elastic-apm-agent-1.43.0.jar &

Properties:

server_url=https://xxxxx.com:8200
recording=false
application_packages=com.example.org

Please let me know if you need any more details.

[Sandeep_Rudra]

Still getting transactions to Kibana from APM Agent.

[Jonas_Kunz]

If I understand you correctly, you are running the attacher again, in order to make the agent "stop". This won't work and rather cause harm, because you'll have two agents running in parallel on the same process.

You instead should do the following:

• When attaching the agent for the first time to the process, provide the --config config_file="/apm-agent-attach/elasticapm.properties" option
  • The properties file should contain recording=true
• When you now want to disable the agent, simply edit the config file to recording=false. The agent will reload the configuration after at most 30 seconds and disable itself. You can reenable it by editing the config file back to recording = true

[Sandeep_Rudra]

Thank you Jonas for quick responses:

When we tried on a new server it started working. The servers where the agent was already running, it did not work. Maybe we need to restart the applications to take effect.

Question, When I run "java -jar ...." The agent is starting but after a while I have to do ctl+c to kill the process. Since it keeps going. Is this normal? If we want to automate the installation process using Ansible, should we install it in a different way?

Also one more, even after we kill the apm process the metrics are still coming in. How does the agent still keep working after killing the apm process?

[Jonas_Kunz]

When we tried on a new server it started working. The servers where the agent was already running, it did not work. Maybe we need to restart the applications to take effect.

As I already said, you must not run the agent twice on the same process!

Question, When I run "java -jar ...." The agent is starting but after a while I have to do ctl+c to kill the process. Since it keeps going. Is this normal? If we want to automate the installation process using Ansible, should we install it in a different way?

Do you mean you start your application via "java -jar ...." or the agent attacher?

Also one more, even after we kill the apm process the metrics are still coming in. How does the agent still keep working after killing the apm process?

There is no separate "apm-process". The attacher forces your application-JVM to load and start the apm agent. The attacher process itself doesn't send any data.

[Sandeep_Rudra]

Thank you for reply Jonas. I got all the answers I need thank you for being so responsive and helpful.