APM and wildfly integration. Couldn't get response from APM agent

catadetest · July 24, 2020, 2:13pm

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 7.1.1:

Elasticsearch version: 7.1.1:

APM Server version: 7.1.1:

APM Agent language and version: elastic-apm-agent-1.18.0.RC1.jar:

Browser version: Chrome 84.0.4147.89 (Official Build) (64-bit):

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

First I've installed the apm-server:

curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-7.1.1-x86_64.rpm
rpm -vi apm-server-7.1.1-x86_64.rpm

I've configured the /etc/apm-server/apm-server.yml with the following details:

apm-server:
host: "0.0.0.0:8200"
rum:
enabled: true

output:
elasticsearch:
hosts: localhost:9200

queue.mem.events: 4096

max_procs: 4

Started the service
Installed the apm agent on java server.
tried the port from java server towards the apm-server and it succeded.
started the wildfly together with the parameters

Fresh install or upgraded from other version?
fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
no

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

The java application is started as a wildfly service: java-wildfly
I have added the following parameters on the /opt/wildfly-bstapp/bin/standalone.conf
export JAVA_OPTS="$JAVA_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=cool-service"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=$JBOSS_MODULES_SYSTEM_PKGS"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://82.78.135.170:8200"

The application is starting, I can even see the above java_opts as parameters listing it's process, however checking the agent status from kibana interface outputs: "No data has been received from agents yet".

Steps to reproduce:

start the wildfly service
browse the application via web broswer
check the Agent status - fails

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
server.log LOG FILE: https://github.com/catadetest/logs/blob/master/server.log

standalone.conf file: https://github.com/catadetest/logs/blob/master/standalone.conf

Any help would be really appreciated
Thanks!

Eyal_Koren · July 26, 2020, 9:00am

Thanks for the question.

Two things I noticed:

Setting the application_packages to $JBOSS_MODULES_SYSTEM_PKGS is incorrect. It should be set with your application's root package/s.
The path to the javaagent - home/catalin/ seems to contain a typo, is that intentional?

If you still cannot find what's wrong, try setting the log_level to DEBUG and upload the log.

catadetest · July 27, 2020, 9:11am

Hi Eyal_Koren,

Not sure what you mean by the step 2 typo. The agent is located in /home/catalin/elastic-apm-agent-1.18.0.RC1.jar

root@web11-server:~# ls -l /home/catalin/elastic-apm-agent-1.18.0.RC1.jar
-rw-rw-r-- 1 catalin catalin 8259183 Jul 24 11:08 /home/catalin/elastic-apm-agent-1.18.0.RC1.jar

I've modified the package name with "com.prompsit.bstweb" as I found it somewhere in the logs, but I'm not sure it's correct. Not sure where to get the application root package name from wildfly.
Please see the modified standalone.conf file:
Also, I've modified the apm-server.yml file to be verbose, but there's no data in /var/log/apm-server:

[root@elk apm-server]# ls -l /var/log/apm-server/
total 0

Also, I've modified the the wildfly logger to be more verbose by setting the "root-logger" to DEBUG in the standalone.xml file.
Now, I have a huge server.log, but I am not able to find any errors related to apm-agent and the Agent status still outputs "No data has been received from agents yet"

Eyal_Koren · July 27, 2020, 9:45am

Sorry, I thought about catalina, disregard.

It should be the root package of the app you are tracing. You would need to figure this out or remove this configuration.

I am looking for the agent logs, not WildFly logs.

I am not even sure the JVM is not even picking the javaagent when it starts.
Try using JAVA_OPTS="$JAVA_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar", meaning - without the export (as it is a conf file and not a script).

catadetest · July 27, 2020, 10:09am

Not sure which is the root package of the app, so I've removed the: JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com.prompsit.bstweb" from the bin/standalone.conf file. However, removing the -Delastic.apm.application_packages will still make the apm-agent to send data? Isn't it a mandatory config line?

Also, I've removed the "export" string and now all of them start with JAVA_OPTS=something instead of export JAVA_OPTS=something.
Here is how it looks:

JAVA_OPTS="$JAVA_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=cool-service"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://82.78.135.170:8200"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_level=DEBUG"

As you can see, Delastic.apm.log_level is set to DEBUG, however I'm not sure where it will write the log, since I can't find anything related in the /var/log for the agent logs.

JVM probably is picking the javaagent, since verifying the process returns the apm javagent in the parameters list:

root@bstweb11-test:/var/log# ps aux | grep -i java

root 63655 25.5 53.6 11390752 5484440 ? Sl 09:54 2:17 /opt/jdk8/bin/java -D[Standalone] -server -Xms2G -Xmx8G -Djboss.bind.address.management=82.78.135.169 -Djboss.management.http.port=9991 -Djboss.bind.address=82.78.135.169 -Djboss.http.port=8082 -Djboss.bind.address.internal=10.0.0.1 -Djboss.http_internal.port=8082 -Djboss.bind.address.ajp_int=127.0.0.1 -Djboss.ajp.port=8011 -Dwebdb.url=jdbc:mysql://10.0.0.15:3306/bstdb?autoReconnect=true&connectTimeout=1000&maxReconnects=2&initialTimeout=1 -Dbst.log.level.console=OFF -Dbst.log.level.file=WARN -Dlog.level.handler.console=OFF -Dsolr.solr.home=/opt/solr-stag/solr -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=250 -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts -verbose:gc -Xloggc:/data/LOGS/javaapp-wildfly/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy -XX:+PrintReferenceGC -Dsun.net.http.allowRestrictedHeaders=true -Dspring.profiles.active=stag -Dcontext_solr_host=82.78.135.166:8081 -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar -Delastic.apm.service_name=cool-service -Delastic.apm.server_urls=http://82.78.135.170:8200 -Delastic.apm.log_level=DEBUG -Dorg.jboss.boot.log.file=/opt/wildfly-javaapp/standalone/log/server.log -Dlogging.configuration=file:/opt/wildfly-javaapp/standalone/configuration/logging.properties -jar /opt/wildfly-javaapp/jboss-modules.jar -mp /opt/wildfly-javaapp/modules org.jboss.as.standalone -Djboss.home.dir=/opt/wildfly-javaapp -Djboss.server.base.dir=/opt/wildfly-javaapp/standalone -c standalone.xml

Eyal_Koren · July 27, 2020, 10:37am

It should default to System.out. Try setting log_file to a specific location.

catadetest · July 27, 2020, 10:54am

Well, this is a bit confusing, as I don't understand how can I set a log level for the apm-agent, since I didn't do an installation at all, but just a jar file which runs at runtime with the java application itself (by wildfly).
Let me provide some more info:
I have 2 servers:
1 - ELK server. It is here where I've installed the apm-server and start it.

2 - java web server. It is here where I downloaded the apm-agent jar file which I passed to wildfly to execute, but it seems the ELK server(where apm-server is installed can't see the agent running)

So as you can see, how can I set a debug level for apm-agent, since there's no installation at all on the operating system.

Thank you so much for your time!
Catalin

Eyal_Koren · July 27, 2020, 12:30pm

When you use the javaagent cmd line directive, the JVM loads the referenced jar when it starts.
Then the agent looks at the System properties you set for server_urls, log_level etc.
What I suggest is that you also set log_file

catadetest · July 27, 2020, 1:07pm

So basicaly I need to add JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_level=DEBUG" in the wildfly bin/standalone.conf file, which I already did.

JAVA_OPTS="$JAVA_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=cool-service"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://137.74.230.166:8200"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_level=DEBUG"

Also, on the apm-server, I set the following:

apm-server:
host: "0.0.0.0:8200"
rum:
enabled: true

output:
elasticsearch:
hosts: localhost:9200

queue.mem.events: 4096

max_procs: 4
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/apm-server
name: apm-server
keepfiles: 7
permissions: 0644

Still, I can't find any erros in the wildfly server.log file

system · August 17, 2020, 9:08am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.