Elasticsearch Rest Client throws java.lang.ArrayStoreException

arony · April 17, 2018, 1:53pm

Hello

I am using Elasticsearch Rest Client v5.5.3 and it throws java.lang.ArrayStoreException after a while application start. The application which uses Elasticsearch rest client executes queries intensively. Would it be the problem? Stack trace is below;

java.lang.ArrayStoreException: org.apache.http.impl.cookie.RFC2965VersionAttributeHandler
	at org.apache.http.impl.cookie.DefaultCookieSpecProvider.create(DefaultCookieSpecProvider.java:92) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:152) ~[flux-core-1.1.0.jar:1.1.0]
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[flux-core-1.1.0.jar:1.1.0]
	at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146) ~[httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:124) ~[httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) ~[httpasyncclient-4.1.2.jar:4.1.2]
	at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:343) ~[rest-5.5.3.jar:5.5.3]
	at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:325) ~[rest-5.5.3.jar:5.5.3]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[rest-5.5.3.jar:5.5.3]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191) ~[rest-5.5.3.jar:5.5.3]
	at com.nova.stats.platform.are.batch.rule.ElasticsearchQueryExecutor.execute(ElasticsearchQueryExecutor.java:36) [classes/:?]

My rest client initialization is in this gist https://gist.github.com/arony/c161dd1f4d4e1cbd34491a0255f50744

cbuescher · April 20, 2018, 10:18am

Thanks for raising this, can you share your applications apache http dependencies? This discussion (java - ArrayStoreException making a post using apache httpasyncclient - Stack Overflow) suggests there might be a mismatch of the httpclient components that get pulled in. I'd like to check if those are the ones the ES rest client uses as transitive dependencies or if there is anything else in the classpath. If we (ES) pull in the dependencies we might have an incompatibility (but that would require further investigation then I think).

arony · April 20, 2018, 11:14am

Thanks for reply, it is certainly httpclient version mismatch with another dependency. After a while I realized that in stack trace exception is coming another dependency that we used. I do not know why but flux-core.1.1.0 (Apache Storm) has dependency httpclient version 4.3.3.

at org.apache.http.impl.cookie.DefaultCookieSpecProvider.create(DefaultCookieSpecProvider.java:92) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:152) ~[flux-core-1.1.0.jar:1.1.0]
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) ~[flux-core-1.1.0.jar:1.1.0]
	at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.2.jar:4.1.2]

Now I am trying to have both version but I could not achieve for now. Below I shared all dependencies we use

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-core</artifactId>
            <version>1.1.0</version>
            <!--<scope>provided</scope>-->
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>flux-core</artifactId>
            <version>1.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>flux-wrappers</artifactId>
            <version>1.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-kafka-client</artifactId>
            <version>1.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.11</artifactId>
            <version>0.10.2.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.zookeeper</groupId>
                    <artifactId>zookeeper</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.googlecode.json-simple</groupId>
            <artifactId>json-simple</artifactId>
            <version>1.1</version>
        </dependency>

        <dependency>
            <groupId>com.nova.stats</groupId>
            <artifactId>stats-common</artifactId>
            <version>1.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.nova.stats</groupId>
            <artifactId>stats-are-batch</artifactId>
            <version>1.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.5</version>
        </dependency>

        <dependency>
            <groupId>com.nova.stats</groupId>
            <artifactId>stats-stix</artifactId>
            <version>1.1</version>
        </dependency>

cbuescher · April 20, 2018, 12:57pm

Hi,

thanks, it looks like you are using maven, could you post the output of mvn dependency:tree here so we can check if any of the verison the ES rest client pulls in are overritten?

arony · April 20, 2018, 1:20pm

Hi,

Thanks, here is what dependency tree looks like

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building stats-first-occurrence 1.1
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ stats-first-occurrence ---
[INFO] com.nova.stats:stats-first-occurrence:jar:1.1
[INFO] +- org.apache.storm:storm-core:jar:1.1.0:compile
[INFO] |  +- com.esotericsoftware:kryo:jar:3.0.3:compile
[INFO] |  |  +- com.esotericsoftware:reflectasm:jar:1.10.1:compile
[INFO] |  |  |  \- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] |  |  +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- org.clojure:clojure:jar:1.7.0:compile
[INFO] |  +- ring-cors:ring-cors:jar:0.1.5:compile
[INFO] |  +- com.lmax:disruptor:jar:3.3.2:compile
[INFO] |  +- org.apache.logging.log4j:log4j-api:jar:2.8:compile
[INFO] |  +- org.apache.logging.log4j:log4j-core:jar:2.8:compile
[INFO] |  +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.8:compile
[INFO] |  +- org.slf4j:log4j-over-slf4j:jar:1.6.6:compile
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] +- org.apache.storm:flux-core:jar:1.1.0:compile
[INFO] |  \- commons-cli:commons-cli:jar:1.2:compile
[INFO] +- org.apache.storm:flux-wrappers:jar:1.1.0:compile
[INFO] |  +- org.apache.storm:multilang-javascript:jar:1.1.0:compile
[INFO] |  +- org.apache.storm:multilang-ruby:jar:1.1.0:compile
[INFO] |  \- org.apache.storm:multilang-python:jar:1.1.0:compile
[INFO] +- org.apache.storm:storm-kafka-client:jar:1.1.0:compile
[INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.2.0:compile
[INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.3:compile
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.8:compile
[INFO] |  +- com.101tec:zkclient:jar:0.10:compile
[INFO] |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] +- org.apache.kafka:kafka-clients:jar:0.10.2.0:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |  \- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] +- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] +- com.nova.stats:stats-common:jar:1.1:compile
[INFO] +- com.nova.stats:stats-are-batch:jar:1.1:compile
[INFO] |  +- com.nova.stats:stats-notifier-mail:jar:1.1:compile
[INFO] |  |  +- javax.mail:mail:jar:1.4.7:compile
[INFO] |  |  |  \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  \- com.nova.stats:stats-notifier-base:jar:1.1:compile
[INFO] |  +- org.quartz-scheduler:quartz:jar:2.2.1:compile
[INFO] |  |  \- c3p0:c3p0:jar:0.9.1.1:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- org.apache.kafka:kafka_2.10:jar:0.10.2.0:compile
[INFO] |  +- org.apache.pdfbox:pdfbox:jar:2.0.5:compile
[INFO] |  |  +- org.apache.pdfbox:fontbox:jar:2.0.5:compile
[INFO] |  |  \- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  +- com.itextpdf:itextpdf:jar:5.5.11:compile
[INFO] |  \- org.elasticsearch.client:rest:jar:5.5.3:compile
[INFO] |     +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] |     +- org.apache.httpcomponents:httpcore:jar:4.4.5:compile
[INFO] |     +- org.apache.httpcomponents:httpasyncclient:jar:4.1.2:compile
[INFO] |     +- org.apache.httpcomponents:httpcore-nio:jar:4.4.5:compile
[INFO] |     \- commons-codec:commons-codec:jar:1.10:compile
[INFO] +- commons-io:commons-io:jar:2.5:compile
[INFO] \- com.nova.stats:stats-stix:jar:1.1:compile
[INFO]    \- com.google.code.gson:gson:jar:2.8.0:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.248 s
[INFO] Finished at: 2018-04-20T16:18:10+03:00
[INFO] Final Memory: 15M/303M
[INFO] ------------------------------------------------------------------------

cbuescher · April 20, 2018, 2:31pm

Thanks, this looks all good to me, at least those are the versions that the ES rest client defines itself.

I looked for infos on compatibility and from the httpasyncclient releasenotes it looks like everything should be okay:

Release 4.1.2
-------------------

This is a maintenance release that fixes a number of issues discovered since 4.1.1 and upgrades
HttpCore and HttpClient dependencies.

Changelog
-------------------

* HttpCore upgraded to version 4.4.5

* HttpClient upgraded to version 4.5.2

I think at this point it would be good if you could narrow done which action in your application is causing the exception to happen so we might be able to reproduce it. You mentioned is only happens after a while, are there any hints what happens before the exception? Is the duration it takes to hit this somehow fixed? Either same time or similar number of requests sent? Any particular type of request that can be identified that might be repeatable to help reproduce?

arony · April 22, 2018, 11:08am

You can reproduce exception by using below two dependencies in any project;

<dependency>
       <groupId>org.apache.storm</groupId>
       <artifactId>flux-core</artifactId>
       <version>1.1.0</version>
</dependency>
<dependency>
       <groupId>org.elasticsearch.client</groupId>
       <artifactId>rest</artifactId>
       <version>5.5.3</version>
 </dependency>

Please have a look this stackoverflow question, Yuriy Alevohin's asnwer. Thanks him, he clearly answered why this exception is been thrown https://stackoverflow.com/questions/49894890/elasticsearch-rest-client-throws-java-lang-arraystoreexception/49898063#49898063

cbuescher · April 23, 2018, 8:48am

Hi @arony,
thanks a lot for digging this up, the pointer to SO confirms that there is a dependency problem. I didn't try to reproduce the issue but looked at the contents of the flux-core 1.1.0 jar, and it indeed contains the entire httpclient codebase (I trust the post that it is version 4.3.3, didn't check).

This is very unfortunate and IMHO bad practice on the flux-core side to embed a commonly used library without shading it. It means you cannot exclude the code from the flux-core jar without taking it apart and repackaging it (which may or may not be possible for you).

One thing you could try is to explicitely exclude the offending httpclient version from the ES rest clients transitive dependencies, but this will potentially create other version conflicts and is entirely untested and unsupported. Welcome to Jar-Hell

I also just found this issue in the Storm repo that suggests that later versions of storm (and with it I believe flux-core), started to shade their external dependencies. You might be able to update to a current version of that library to avoid the conflict.

Let me know if this helps and what you ended up doing.

arony · April 25, 2018, 6:32pm

Hi @cbuescher,

Finally, I downloaded flux-core source and excluded httpclient 4.3.3. The problem solved! Thanks for help

cbuescher · April 25, 2018, 8:07pm

Just asking for future reference (if anybody happens to run into the same issue): this sound like you manually pruned the httpclient from the flux-core jar? Or did you go with excluding the transitive dependency of the ES rest client?

arony · April 27, 2018, 7:04am

I manually pruned httpclient and httpcore from flux-core since we do not need them

<configuration>
    <transformers>
         <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
            <mainClass>org.apache.storm.flux.Flux</mainClass>
       </transformer>
   </transformers>
   <artifactSet>
       <excludes>
           <exclude>org.apache.httpcomponents:httpclient</exclude>
           <exclude>org.apache.httpcomponents:httpcore</exclude>
       </excludes>
   </artifactSet>
</configuration>

system · May 25, 2018, 7:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.