I am using Elasticsearch Rest Client v5.5.3 and it throws java.lang.ArrayStoreException after a while application start. The application which uses Elasticsearch rest client executes queries intensively. Would it be the problem? Stack trace is below;
java.lang.ArrayStoreException: org.apache.http.impl.cookie.RFC2965VersionAttributeHandler
at org.apache.http.impl.cookie.DefaultCookieSpecProvider.create(DefaultCookieSpecProvider.java:92) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:152) ~[flux-core-1.1.0.jar:1.1.0]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[flux-core-1.1.0.jar:1.1.0]
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:124) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:343) ~[rest-5.5.3.jar:5.5.3]
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:325) ~[rest-5.5.3.jar:5.5.3]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[rest-5.5.3.jar:5.5.3]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191) ~[rest-5.5.3.jar:5.5.3]
at com.nova.stats.platform.are.batch.rule.ElasticsearchQueryExecutor.execute(ElasticsearchQueryExecutor.java:36) [classes/:?]
Thanks for raising this, can you share your applications apache http dependencies? This discussion (java - ArrayStoreException making a post using apache httpasyncclient - Stack Overflow) suggests there might be a mismatch of the httpclient components that get pulled in. I'd like to check if those are the ones the ES rest client uses as transitive dependencies or if there is anything else in the classpath. If we (ES) pull in the dependencies we might have an incompatibility (but that would require further investigation then I think).
Thanks for reply, it is certainly httpclient version mismatch with another dependency. After a while I realized that in stack trace exception is coming another dependency that we used. I do not know why but flux-core.1.1.0 (Apache Storm) has dependency httpclient version 4.3.3.
at org.apache.http.impl.cookie.DefaultCookieSpecProvider.create(DefaultCookieSpecProvider.java:92) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:152) ~[flux-core-1.1.0.jar:1.1.0]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[flux-core-1.1.0.jar:1.1.0]
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.2.jar:4.1.2]
Now I am trying to have both version but I could not achieve for now. Below I shared all dependencies we use
thanks, it looks like you are using maven, could you post the output of mvn dependency:tree here so we can check if any of the verison the ES rest client pulls in are overritten?
Thanks, this looks all good to me, at least those are the versions that the ES rest client defines itself.
I looked for infos on compatibility and from the httpasyncclient releasenotes it looks like everything should be okay:
Release 4.1.2
-------------------
This is a maintenance release that fixes a number of issues discovered since 4.1.1 and upgrades
HttpCore and HttpClient dependencies.
Changelog
-------------------
* HttpCore upgraded to version 4.4.5
* HttpClient upgraded to version 4.5.2
I think at this point it would be good if you could narrow done which action in your application is causing the exception to happen so we might be able to reproduce it. You mentioned is only happens after a while, are there any hints what happens before the exception? Is the duration it takes to hit this somehow fixed? Either same time or similar number of requests sent? Any particular type of request that can be identified that might be repeatable to help reproduce?
Hi @arony,
thanks a lot for digging this up, the pointer to SO confirms that there is a dependency problem. I didn't try to reproduce the issue but looked at the contents of the flux-core 1.1.0 jar, and it indeed contains the entire httpclient codebase (I trust the post that it is version 4.3.3, didn't check).
This is very unfortunate and IMHO bad practice on the flux-core side to embed a commonly used library without shading it. It means you cannot exclude the code from the flux-core jar without taking it apart and repackaging it (which may or may not be possible for you).
One thing you could try is to explicitely exclude the offending httpclient version from the ES rest clients transitive dependencies, but this will potentially create other version conflicts and is entirely untested and unsupported. Welcome to Jar-Hell
I also just found this issue in the Storm repo that suggests that later versions of storm (and with it I believe flux-core), started to shade their external dependencies. You might be able to update to a current version of that library to avoid the conflict.
Let me know if this helps and what you ended up doing.
Just asking for future reference (if anybody happens to run into the same issue): this sound like you manually pruned the httpclient from the flux-core jar? Or did you go with excluding the transitive dependency of the ES rest client?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
