# APM and wildfly integration. Couldn't get response from APM agent **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [July 24, 2020, 2:13pm UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521 "2020-07-24T14:13:52Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [July 24, 2020, 2:13pm UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/1 "2020-07-24T14:13:53Z")
If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂 TIP 1: select at least one tag that further categorizes your topic. For example `server` for APM Server related questions, `java` for questions regarding the Elastic APM Java agent, or `ui` for questions about the APM App within Kibana. TIP 2: Check out the [troubleshooting guide](https://www.elastic.co/guide/en/apm/get-started/current/troubleshooting.html) first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you. **Kibana version: 7.1.1** : **Elasticsearch version: 7.1.1** : **APM Server version: 7.1.1** : **APM Agent language and version: elastic-apm-agent-1.18.0.RC1.jar** : **Browser version: Chrome 84.0.4147.89 (Official Build) (64-bit)**: **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: 1. First I've installed the apm-server: - curl -L -O [https://artifacts.elastic.co/downloads/apm-server/apm-server-7.1.1-x86\_64.rpm](https://artifacts.elastic.co/downloads/apm-server/apm-server-7.1.1-x86_64.rpm) - rpm -vi apm-server-7.1.1-x86\_64.rpm 1. I've configured the /etc/apm-server/apm-server.yml with the following details: apm-server: host: "0.0.0.0:8200" rum: enabled: true output: elasticsearch: hosts: localhost:9200 queue.mem.events: 4096 max\_procs: 4 1. Started the service 2. Installed the apm agent on java server. 3. tried the port from java server towards the apm-server and it succeded. 4. started the wildfly together with the parameters **Fresh install or upgraded from other version?** fresh install **Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. no **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: The java application is started as a wildfly service: java-wildfly I have added the following parameters on the /opt/wildfly-bstapp/bin/standalone.conf export JAVA\_OPTS="$JAVA\_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar" export JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.service\_name=cool-service" export JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.application\_packages=$JBOSS\_MODULES\_SYSTEM\_PKGS" export JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.server\_urls=http://82.78.135.170:8200" The application is starting, I can even see the above java\_opts as parameters listing it's process, however checking the agent status from kibana interface outputs: "No data has been received from agents yet". **Steps to reproduce** : 1. start the wildfly service 2. browse the application via web broswer 3. check the Agent status - fails **Errors in browser console (if relevant)**: **Provide logs and/or server output (if relevant)**: server.log LOG FILE: [https://github.com/catadetest/logs/blob/master/server.log](https://github.com/catadetest/logs/blob/master/server.log) standalone.conf file: [https://github.com/catadetest/logs/blob/master/standalone.conf](https://github.com/catadetest/logs/blob/master/standalone.conf) Any help would be really appreciated Thanks! ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [July 26, 2020, 9:00am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/2 "2020-07-26T09:00:40Z")
Thanks for the question. Two things I noticed: 1. Setting the [`application_packages`](https://www.elastic.co/guide/en/apm/agent/java/current/config-stacktrace.html#config-application-packages) to `$JBOSS_MODULES_SYSTEM_PKGS` is incorrect. It should be set with your application's root package/s. 2. The path to the `javaagent` - `home/catalin/` seems to contain a typo, is that intentional? If you still cannot find what's wrong, try setting the [`log_level`](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-level) to `DEBUG` and upload the log. ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [July 27, 2020, 9:11am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/3 "2020-07-27T09:11:04Z")
Hi Eyal\_Koren, Not sure what you mean by the step 2 typo. The agent is located in /home/catalin/elastic-apm-agent-1.18.0.RC1.jar ```auto root@web11-server:~# ls -l /home/catalin/elastic-apm-agent-1.18.0.RC1.jar -rw-rw-r-- 1 catalin catalin 8259183 Jul 24 11:08 /home/catalin/elastic-apm-agent-1.18.0.RC1.jar ``` I've modified the package name with "com.prompsit.bstweb" as I found it somewhere in the logs, but I'm not sure it's correct. Not sure where to get the application root package name from wildfly. Please see the modified [standalone.conf](https://github.com/catadetest/logs/blob/master/standalone.conf) file: Also, I've modified the [apm-server.yml](https://github.com/catadetest/logs/blob/master/apm-server.yml) file to be verbose, but there's no data in /var/log/apm-server: ```auto [root@elk apm-server]# ls -l /var/log/apm-server/ total 0 ``` Also, I've modified the the wildfly logger to be more verbose by setting the "root-logger" to DEBUG in the standalone.xml file. Now, I have a huge [server.log](https://github.com/catadetest/logs/blob/master/server.log), but I am not able to find any errors related to apm-agent and the Agent status still outputs "No data has been received from agents yet" ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [July 27, 2020, 9:45am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/4 "2020-07-27T09:45:25Z")
> [@catadetest](#): > > Not sure what you mean by the step 2 typo. The agent is located in /home/catalin/elastic-apm-agent-1.18.0.RC1.jar Sorry, I thought about `catalina`, disregard. > [@catadetest](#): > > I've modified the package name with "com.prompsit.bstweb" as I found it somewhere in the logs, but I'm not sure it's correct. Not sure where to get the application root package name from wildfly. It should be the root package of the app you are tracing. You would need to figure this out or remove this configuration. > [@catadetest](#): > > Also, I've modified the the wildfly logger to be more verbose by setting the "root-logger" to DEBUG in the standalone.xml file. I am looking for the agent logs, not WildFly logs. I am not even sure the JVM is not even picking the `javaagent` when it starts. Try using `JAVA_OPTS="$JAVA_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar"`, meaning - without the `export` (as it is a conf file and not a script). ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [July 27, 2020, 10:09am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/5 "2020-07-27T10:09:17Z")
Not sure which is the root package of the app, so I've removed the: JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.application\_packages=com.prompsit.bstweb" from the bin/standalone.conf file. However, removing the -Delastic.apm.application\_packages will still make the apm-agent to send data? Isn't it a mandatory config line? Also, I've removed the "export" string and now all of them start with **JAVA\_OPTS=something** instead of **export JAVA\_OPTS=something**. Here is how it looks: ```auto JAVA_OPTS="$JAVA_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar" JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=cool-service" JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://82.78.135.170:8200" JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_level=DEBUG" ``` As you can see, Delastic.apm.log\_level is set to DEBUG, however I'm not sure where it will write the log, since I can't find anything related in the /var/log for the agent logs. JVM probably is picking the javaagent, since verifying the process returns the apm javagent in the parameters list: > root@bstweb11-test:/var/log# ps aux | grep -i java > root 63655 25.5 53.6 11390752 5484440 ? Sl 09:54 2:17 /opt/jdk8/bin/java -D[Standalone] -server -Xms2G -Xmx8G -Djboss.bind.address.management=82.78.135.169 -Djboss.management.http.port=9991 -Djboss.bind.address=82.78.135.169 -Djboss.http.port=8082 -Djboss.bind.address.internal=10.0.0.1 -Djboss.http\_internal.port=8082 -Djboss.bind.address.ajp\_int=127.0.0.1 -Djboss.ajp.port=8011 -Dwebdb.url=jdbc:mysql://10.0.0.15:3306/bstdb?autoReconnect=true&connectTimeout=1000&maxReconnects=2&initialTimeout=1 -Dbst.log.level.console=OFF -Dbst.log.level.file=WARN -Dlog.level.handler.console=OFF -Dsolr.solr.home=/opt/solr-stag/solr -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=250 -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts -verbose:gc -Xloggc:/data/LOGS/javaapp-wildfly/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy -XX:+PrintReferenceGC -Dsun.net.http.allowRestrictedHeaders=true -Dspring.profiles.active=stag -Dcontext\_solr\_host=82.78.135.166:8081 -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar -Delastic.apm.service\_name=cool-service -Delastic.apm.server\_urls=[http://82.78.135.170:8200](http://82.78.135.170:8200) -Delastic.apm.log\_level=DEBUG -Dorg.jboss.boot.log.file=/opt/wildfly-javaapp/standalone/log/server.log -Dlogging.configuration=file:/opt/wildfly-javaapp/standalone/configuration/logging.properties -jar /opt/wildfly-javaapp/jboss-modules.jar -mp /opt/wildfly-javaapp/modules org.jboss.as.standalone -Djboss.home.dir=/opt/wildfly-javaapp -Djboss.server.base.dir=/opt/wildfly-javaapp/standalone -c standalone.xml ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [July 27, 2020, 10:37am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/6 "2020-07-27T10:37:39Z")
> [@catadetest](#): > > I'm not sure where it will write the log, since I can't find anything related in the /var/log for the agent logs It should default to `System.out`. Try setting [`log_file`](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-file) to a specific location. ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [July 27, 2020, 10:54am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/7 "2020-07-27T10:54:13Z")
Well, this is a bit confusing, as I don't understand how can I set a log level for the apm-agent, since I didn't do an installation at all, but just a jar file which runs at runtime with the java application itself (by wildfly). Let me provide some more info: I have 2 servers: 1 - ELK server. It is here where I've installed the apm-server and start it. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/8/58351b96ff47ab3dbf2b76d691a0ebd2f3d41071.png) 2 - java web server. It is here where I downloaded the apm-agent jar file which I passed to wildfly to execute, but it seems the ELK server(where apm-server is installed can't see the agent running) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/0/4044ea3b2b0a93acb1c687a2038346bc89e888a2.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/3/b312c9e259c2a3c085392ac81b58cc88f41d21ac.png) So as you can see, how can I set a debug level for apm-agent, since there's no installation at all on the operating system. Thank you so much for your time! Catalin ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [July 27, 2020, 12:30pm UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/8 "2020-07-27T12:30:16Z")
When you use the `javaagent` cmd line directive, the JVM loads the referenced jar when it starts. Then the agent looks at the System properties you set for `server_urls`, `log_level` etc. What I suggest is that you also set `log_file` ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [July 27, 2020, 1:07pm UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/9 "2020-07-27T13:07:26Z")
So basicaly I need to add JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.log\_level=DEBUG" in the wildfly bin/standalone.conf file, which I already did. > JAVA\_OPTS="$JAVA\_OPTS -javaagent:/home/catalin/elastic-apm-agent-1.18.0.RC1.jar" > JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.service\_name=cool-service" > JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.server\_urls=[http://137.74.230.166:8200](http://137.74.230.166:8200)" > JAVA\_OPTS="$JAVA\_OPTS -Delastic.apm.log\_level=DEBUG" Also, on the apm-server, I set the following: > apm-server: > host: "0.0.0.0:8200" > rum: > enabled: true > > output: > elasticsearch: > hosts: localhost:9200 > > queue.mem.events: 4096 > > max\_procs: 4 > logging.level: info > logging.to\_files: true > logging.files: > path: /var/log/apm-server > name: apm-server > keepfiles: 7 > permissions: 0644 Still, I can't find any erros in the wildfly server.log file ☹ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 17, 2020, 9:08am UTC](https://discuss.elastic.co/t/apm-and-wildfly-integration-couldnt-get-response-from-apm-agent/242521/10 "2020-08-17T09:08:51Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.