Application is down when run a few mintues

Elastic Observability APM
1

Java Agent Version:1.18.1
JDK Version:1.7.79
APM Server Version:7.4.0
JVM args:

export JAVA_OPTS="$JAVA_OPTS -javaagent:elastic-apm-agent-1.18.1.jar"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=jboss-service"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com.*"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://x.x.x.x:8200,http://x.x.x.x:8200"

When run about two mintues every time,the application is down without any log.

2

Hi @hyhong,

  • Can you provide the full JVM arguments list ?
  • Are you using g1gc garbage collector algorithm ?
  • Did you upgraded from a previous agent version (like 1.16.x or 1.17.x) ?

There should be some error logs files with name hs_err_pid_xxxx.log in the current working directory or /tmp that could help provide some extra information about the crash.

3
  • elasticapm.properties：
recording=true
transaction_sample_rate=0.01
transaction_max_spans=500
circuit_breaker_enabled=true
stress_monitoring_interval= 10s
stress_monitor_gc_stress_threshold=0.9
stress_monitor_gc_relief_threshold=0.75
stress_monitor_cpu_duration_threshold=1m
stress_monitor_system_cpu_stress_threshold=0.9
stress_monitor_system_cpu_relief_threshold=0.75
capture_headers=false
stack_trace_limit=0
span_frames_min_duration=0ms
max_queue_size=256
4

Could you upload the full log file instead of screenshots of it ? If there are sensitive values in it you could also send it to me as a private message.

As the issue seems quite easy to reproduce within a few minutes, did it happen multiple times and were you able to systematically reproduce it ?

Would it be possible to try with a more recent version of Jdk7 ?
Unfortunately Oracle does not provide binaries, but there are other vendors that do:

It looks like a very similar issue that you already reported here, but at the time it was with an older version of the agent 1.16.0, does this one also only happens in production ? Also, is there any custom plugin involved ?

Is there any kind of load on the application when the issue happens ? If the issue only happens in production or for example during a load test, the reason this does not happen elsewhere it might be related to that.

5

Yes,the 1.16.0 happen the same as 1.18.1.Now it's happen in production and test without custom plugin,only elastic-apm-agent.1.18.1.jar.

6

I have tried put it in githup:
https://github.com/koala775/hs_err_pid39230

7

It seems that you have two copies of the agent in your JVM arguments: -javaagent:/opt/app/elk_6/elastic-ap↪m-agent-1.18.1.jar -javaagent:/opt/app/elk_6/apm-agent-java-custom-1.2.jar

One seems to be the 1.18.1 release, while the other appears to be a custom fork.

If you use your own build, you should ensure that it's based on either master or 1.18.1 release and only include one copy of the agent in JVM arguments.

8

The apm-agent-java-custom-1.2.jar is implement by bytebuddy+java public api

9

Could you try without the custom agent and only use Elastic agent to see if that makes a difference ?
Also, which version of bytebuddy is used within the custom agent ?

10

Yes,I have tried with just Elastic agent.The phenomenon is the same as , the version 1.15.0, 1.16.0, 1.18.1 all I have tried. The custom agent's bytebuddy is used with the same as Elastic agent's version.
Also , the custom agent have some suggestions?

11

Ok, so the other agent seems not related, did you tried with a more recent JDK version ? As noted before there are other vendors that provide up-to-date versions for Java7.

Then, using another agent to use the public API is way over-complicated, directly using the API from within the application should be preferred. There is no dependency between API and the agent as the API is a no-op dependency (it just does nothing), the implementation is provided at runtime when deployed with the agent.

12

We use custom agent beause we need use public api to add in third jar's method.The aim to no invasion.So,there some effect if use two agent.

13

The jdk version we haven't try beause in product.

14

Ok, I understand the use case with the second agent then, as you can't modify the application directly.

We will provide a better alternative to do that in the future with using external plugins, we have such an example plugin here, but it's not really well documented and ready for prime time yet.

Going back to your JDK crash issue, trying to update it is currently the best option.

15

Ok，I will try to upgrade.Thanks.

16

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.