APM Java Agent getting not all data

Elastic Observability APM
1

Hi all,

I'm new here and hopefully using the forum right. I set up an ELK stack with beats version 7.11.1 newly in a local environment via Ansible and Vagrant. Everything is working fine.

Now I wanted to collect and visualize JMX data using APM. I wrote a role for the APM server, which works. In Kibana I get a message that I have set up the APM Server correctly. Then I downloaded the APM Java Agent as described in the instructions to the node from which I want to get the data. I wrote an elasticapm.properties. It is located in the folder where the .jar file of the agent is located.

When I start the application I get the following parameters:

   -javaagent:/home/tracey/elastic-apm-agent-1.21.0.jar \
   -Delastic.apm.server_urls=http://localhost:8200 \
   -Delastic.apm.application_packages=<name> \

I get data displayed in Kibana under APM, but the following message at agent status: "No data has been received from agents yet". Could it be that this is why I am not getting all the data from each worker displayed under APM? Also, I wanted to display the JMX data of the application, I just don't know how. Can you please explain where and how I can do that or maybe I am not getting the data due to the message about the status of the agent?

2

Hi @Katharina , welcome to the forum !

Can you provide a screenshot where you see the 'No data has been received from agents yet' message ? Does the message is still there once your JVM has processed a few transactions ?

Regarding JMX, agent capture JMX beans values as metrics by default, thus you can see the graphs in the APM / Services / <your service name> section. Those JVM metrics are documented here.

If you want to capture more than what we have by default, you will have to configure it through the JMX configuration.

The high level steps are:

  • find the relevant JMX beans you want to capture, using jconsole or visualvm tools with a running JVM
  • edit agent configuration to make it capture those metrics
  • check in Kibana / Discover tab that you get your captured metrics, go back to 1st step if it does not

Since there might be some trial & error involved due to JMX beans syntax, using an external .properties configuration file for agent configuration could help to make it easier without restarting the JVM (agent will detect configuration change without restart). This configuration dynamic configuration option is not yet available directly from Kibana ([Configuration | APM Java Agent Reference [1.x] | Elastic](https://configuration sources))

3

Hi @Sylvain_Juge,

thank you very much for your message. Here is the screenshot you have asked for:

AgentStatus
AgentStatus1759×940 109 KB

The agent status does not change, even after transactions have been received and displayed in the “APM | Services” Tab.

I have written an elasticapm.properties file to try to get JMX metrics with "capture_jmx_metrics". Unfortunately, this does not work. The agent configuration didn't work at all for me using the properties file. That's why I do the launching and configuration of the agent directly in the application using the -javaagent flag.

We found out that we get all the data displayed if we run only the webWorker process that uses HTTP. Unfortunately, we don't get the data of the other worker processes doing background work (e.g., long running background tasks) displayed then. But I would like to see all data of the worker processes in the APM overview and also the single beans.

In the overview of the APM I get this displayed:

APMwebWorker
APMwebWorker1426×579 26.3 KB

However, I would like to have the other workers displayed in this overview, such as the jobWorker. I get the data if I enter jobWorker in the URL instead of webWorker, for example:

http://10.211.55.11:5601/app/apm/services/jobWorker/transactions/...

jobWorkerAPM2

But if I run all workers, I can't show all transactions either. I only get them when the webWorker is running alone. Unfortunately this is not an option, because all workers have to be running for the application to work. I would therefore like to be able to display all the data from all the workers and JMX metrics that we need.

Do you have any idea how I can display all the data? Do the services only show those using HTTP requests? If yes, how to display JMX beans from background processes not directly related to an HTTP request?

4

I think that the status reported might be misleading as if you have data the agent is properly working and sending data. That's probably a UI bug, but not something blocking here.

In order to debug what is wrong with configuration here, the best way to do that is to set log_level=debug and check the configuration that is read by the agent when the JVM starts. Also, using -javaagent option is not exclusive to using external properties file.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.