com.lmax.disruptor.IgnoreExceptionHandler - Exception processing

APM Agent 1.51
Elastic Stack 8.11.4
Java 8
Weblogic 12.2

We are using the APM Java agent and attaching it programatically using "ElasticApmAttacher.attach()".

It is running more then a year now but the last months we are experiencing problems.
Java Agent is not able to send data to Elasticsearch. The following exception is found thousands of times in the weblogic log:

[elastic-apm-server-reporter] INFO com.lmax.disruptor.IgnoreExceptionHandler - Exception processing: 112600405 Type: null java.lang.NullPointerException: null

After restart of the weblogic server the proces continues but very soon (sometimes a couple of hours sometimes a couple of days later) the same issue appears with the same error log.

I could not find any clues searching the internet.

Any ideas? Where to start to analyse this problem?

Best regards,

Murat

Hi,

Thanks for reporting this issue.

Could you provide a copy of the whole stack trace please ? Sent is as DM if needed.

Do you have an estimate of the frequency this issue is happening ? is that one or twice per month or more weekly/daily ?

Thanks for the quick reply!

I don't have a stack trace yet. Not showing up in the weblogic logs.
But I have requested the Elasticsearch logs from the operators. This could take a couple of days unfortunately.

Should I also have the Kibana logs?

It started to happen more frequently the last 3 months. Last time, this happened within hours after restarting weblogic.
A couple of months ago it worked for some weeks after a restart.

When it happens it is constantly throwing this error thousands of times per day.

The Elasticsearch and Kibana logs are not relevant here, what you need to provide is the elastic apm agent logs.

A common procedure to capture those in a separate file is available here.

Given you are deploying on Weblogic, does it means one webapp deployed on the server deploys the agent through runtime attach with the API, and then it instruments the whole service ?

As an alternative, trying to use the more simple -javaagent setup method could likely help you achieve better stability. While we haven't changed this part recently it might still have a few corner cases.

Okay understood.

I will configure APM via properties to log to a file and will set the log_level to debug for a short time at runtime in production.

Thank you!

Kind regards,
Murat