# Application is down when run a few mintues **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [October 23, 2020, 6:21am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087 "2020-10-23T06:21:04Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 23, 2020, 6:21am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/1 "2020-10-23T06:21:05Z")
Java Agent Version:1.18.1 JDK Version:1.7.79 APM Server Version:7.4.0 JVM args: ```auto export JAVA_OPTS="$JAVA_OPTS -javaagent:elastic-apm-agent-1.18.1.jar" export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=jboss-service" export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com.*" export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://x.x.x.x:8200,http://x.x.x.x:8200" ``` When run about two mintues every time,the application is down without any log. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 26, 2020, 7:41am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/2 "2020-10-26T07:41:16Z")
Hi @hyhong, - Can you provide the full JVM arguments list ? - Are you using g1gc garbage collector algorithm ? - Did you upgraded from a previous agent version (like 1.16.x or 1.17.x) ? There should be some error logs files with name `hs_err_pid_xxxx.log` in the current working directory or `/tmp` that could help provide some extra information about the crash. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 26, 2020, 8:11am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/3 "2020-10-26T08:11:29Z")
- elasticapm.properties： ```auto recording=true transaction_sample_rate=0.01 transaction_max_spans=500 circuit_breaker_enabled=true stress_monitoring_interval= 10s stress_monitor_gc_stress_threshold=0.9 stress_monitor_gc_relief_threshold=0.75 stress_monitor_cpu_duration_threshold=1m stress_monitor_system_cpu_stress_threshold=0.9 stress_monitor_system_cpu_relief_threshold=0.75 capture_headers=false stack_trace_limit=0 span_frames_min_duration=0ms max_queue_size=256 ``` - I don't use g1gc - I haven't upgrade. - The log is: ![1](https://us1.discourse-cdn.com/elastic/original/3X/f/2/f2b8b9b48f24ffbb443de284d654b2ad68bb4887.png) ![2](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b76d48d39697328d2d4cbc945b5837b13446da05.png) ![3](https://us1.discourse-cdn.com/elastic/original/3X/5/1/5124852f0fbf1841ab8765c531f7e5f44a3ea9b0.png) ![4](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1a911584f96617492c378b9dd7e0107a944a8d3c.png) ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 26, 2020, 1:03pm UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/4 "2020-10-26T13:03:18Z")
Could you upload the full log file instead of screenshots of it ? If there are sensitive values in it you could also send it to me as a private message. As the issue seems quite easy to reproduce within a few minutes, did it happen multiple times and were you able to systematically reproduce it ? Would it be possible to try with a more recent version of Jdk7 ? Unfortunately Oracle does not provide binaries, but there are other vendors that do: - [https://packages.debian.org/jessie/openjdk-7-jdk](https://packages.debian.org/jessie/openjdk-7-jdk) is available for debian-based systems. - Azul Zulu VM also provides a suitable version: [https://www.azul.com/downloads/zulu-community/?architecture=x86-64-bit&package=jdk](https://www.azul.com/downloads/zulu-community/?architecture=x86-64-bit&package=jdk) It looks like a very similar issue that you already reported [here](https://discuss.elastic.co/t/application-is-down/252309), but at the time it was with an older version of the agent 1.16.0, does this one also only happens in production ? Also, is there any custom plugin involved ? Is there any kind of load on the application when the issue happens ? If the issue only happens in production or for example during a load test, the reason this does not happen elsewhere it might be related to that. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 27, 2020, 4:54am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/5 "2020-10-27T04:54:08Z")
Yes,the 1.16.0 happen the same as 1.18.1.Now it's happen in production and test without custom plugin,only elastic-apm-agent.1.18.1.jar. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 27, 2020, 5:30am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/6 "2020-10-27T05:30:11Z")
I have tried put it in githup: [https://github.com/koala775/hs\_err\_pid39230](https://github.com/koala775/hs_err_pid39230) ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 27, 2020, 7:32am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/7 "2020-10-27T07:32:59Z")
It seems that you have two copies of the agent in your JVM arguments: `-javaagent:/opt/app/elk_6/elastic-ap↪m-agent-1.18.1.jar -javaagent:/opt/app/elk_6/apm-agent-java-custom-1.2.jar` One seems to be the 1.18.1 release, while the other appears to be a custom fork. If you use your own build, you should ensure that it's based on either `master` or `1.18.1` release and only include one copy of the agent in JVM arguments. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 27, 2020, 7:54am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/8 "2020-10-27T07:54:03Z")
The apm-agent-java-custom-1.2.jar is implement by bytebuddy+java public api ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 27, 2020, 8:13am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/9 "2020-10-27T08:13:34Z")
Could you try without the custom agent and only use Elastic agent to see if that makes a difference ? Also, which version of bytebuddy is used within the custom agent ? ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 27, 2020, 8:34am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/10 "2020-10-27T08:34:58Z")
Yes,I have tried with just Elastic agent.The phenomenon is the same as , the version 1.15.0, 1.16.0, 1.18.1 all I have tried. The custom agent's bytebuddy is used with the same as Elastic agent's version. Also , the custom agent have some suggestions? ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 27, 2020, 8:38am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/11 "2020-10-27T08:38:11Z")
Ok, so the other agent seems not related, did you tried with a more recent JDK version ? As noted before there are other vendors that provide up-to-date versions for Java7. Then, using another agent to use the public API is way over-complicated, directly using the API from within the application should be preferred. There is no dependency between API and the agent as the API is a no-op dependency (it just does nothing), the implementation is provided at runtime when deployed with the agent. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 27, 2020, 9:09am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/12 "2020-10-27T09:09:03Z")
We use custom agent beause we need use public api to add in third jar's method.The aim to no invasion.So,there some effect if use two agent. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 27, 2020, 9:29am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/13 "2020-10-27T09:29:37Z")
The jdk version we haven't try beause in product. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [October 29, 2020, 8:17am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/14 "2020-10-29T08:17:28Z")
Ok, I understand the use case with the second agent then, as you can't modify the application directly. We will provide a better alternative to do that in the future with using external plugins, we have such an [example plugin here](https://github.com/elastic/apm-agent-java/tree/master/integration-tests/external-plugin-test), but it's not really well documented and ready for prime time yet. Going back to your JDK crash issue, trying to update it is currently the best option. ---
**Author:** ![hyhong](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hyhong](https://discuss.elastic.co/u/hyhong)\ **Post date:** [October 29, 2020, 8:25am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/15 "2020-10-29T08:25:41Z")
Ok，I will try to upgrade.Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 19, 2020, 4:25am UTC](https://discuss.elastic.co/t/application-is-down-when-run-a-few-mintues/253087/16 "2020-11-19T04:25:47Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.