Weird exceptions from apm agent

[ciment]

Hello,

I use ES 7.6.1. with APM Server 7.6.1 and apm-java agent 1.14.0.

In my jakarta application, when I`m fetching data from ES via RestHighLevelClient I get some exception to server.log

java.lang.ClassCastException: co.elastic.apm.agent.es.restclient.ElasticsearchRestClientInstrumentationHelperImpl cannot be cast to co.elastic.apm.agent.es.restclient.ElasticsearchRestClientInstrumentationHelper
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:234)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:970)

or

java.lang.ClassCastException: co.elastic.apm.agent.httpclient.helper.RequestHeaderAccessor cannot be cast to co.elastic.apm.agent.impl.transaction.TextHeaderSetter
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:244)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:970)

Finally, I`m able to fetch data, but exceptions are little bit annoying.

How can I fix these problems?

[nugusbayevkk (nugusbayevk)]

Hi, what version of RestHighLevelClient do you use?

[ciment]

also the same 7.6.1

[nugusbayevkk (nugusbayevk)]

@ciment,
I wrote application with spring boot and tested with that versions, but cannot reproduce problem.
What application server do you use?
Can you share example-application where the problem occurs?

[ciment]

@nugusbayevkk

I use payara 5.201.

I found where the problem is. I use JRebel agent for hot redeploys. When I start payara with JRebel configuration, I see these exceptions. Without JRebel no exceptions.

Thanks @nugusbayevkk for your investigation. It helps me to find problem somewhere else.

[ciment]

I will forward the problem to JRebel developers...

[Sylvain_Juge]

Hi @ciment,

Our agent have to use some non-standard classloader delegation because some agent classes are loaded in bootstrap classloader (that does not know about application classes), and in order to access application classes like Servlets or other frameworks, some classes accessed by instrumented code need to be in application classloader. As a result, any other agent that re-transforms without applying this same strategy may trigger side effects like this.