java.lang.ClassNotFoundException: org.elasticsearch.search.aggregations.metrics.cardinality.ParsedCardinality

iti · August 7, 2019, 6:11pm

My environment: two ES clusters: one ES 6.7 and one ES 7.1

My app need to use High Level REST Client to access both cluster at the same time.

When I am using 7.x REST CLIENT with 7.X ES library. When accessing 6.x ES, I got Ccs_minimize_roundtrips error when using 7.X high level REST API to access 6.x index

When I am using 6.x REST CLient with 7.X ES, when accessing 7.x ES,I got java.lang.ClassNotFoundException: org.elasticsearch.search.aggregations.metrics.cardinality.ParsedCardinality since I have a Cardinality Aggregation.

When I am using both 6.x REST Client and ES lib, when access 7.x ES, some of the aggregation result on 7.X ES is not correct.

Any idea how to solve this issue? Thanks

javanna · August 12, 2019, 8:52am

Hi,
this one looks like a bug at first glance. Do you have the complete stacktrace of that exception? And steps to reproduce it?

iti · August 14, 2019, 12:07am

Here is the stack trace:
ElasticsearchStatusException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [xxx-api]]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2053)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2030)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092)
	...
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [xxx], URI [/xxx-api/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [xxx-api]","resource.type":"index_or_alias","resource.id":"xxx-api","index_uuid":"_na_","index":"xxx-api"}],"type":"index_not_found_exception","reason":"no such index [xxx-api]","resource.type":"index_or_alias","resource.id":"xxx-api","index_uuid":"_na_","index":"xxx-api"},"status":404}
		at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
		... 113 more
	Caused by: org.elasticsearch.client.ResponseException: method [POST], host [xxx], URI [/xxx-api/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [xxx-api]","resource.type":"index_or_alias","resource.id":"xxx-api","index_uuid":"_na_","index":"xxx-api"}],"type":"index_not_found_exception","reason":"no such index [xxx-api]","resource.type":"index_or_alias","resource.id":"xxx-api","index_uuid":"_na_","index":"xxx-api"},"status":404}
		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:552)
		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
		at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123)
		at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
		at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
		at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
		at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
		at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
		at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
		at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
		at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
		at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
		at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
		at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
		at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
		at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
		... 1 more
2019-08-13 16:36:13.750 DEBUG 4688 --- [  XNIO-2 task-1] xxx.errors.ExceptionTranslator   : An unexpected error occurred: null

java.lang.NullPointerException: null
	at xxx.dao.impl.QueryService.search(QueryService.java:237)
	at xxx.dao.impl.QueryService.search(QueryService.java:100)
	at xxx.web.rest.SearchResource.getSummary(SearchResource.java:75)
	at xxx.web.rest.SearchResource$$FastClassBySpringCGLIB$$cec506a1.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	...
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2019-08-13 16:36:13.788  WARN 4688 --- [  XNIO-2 task-1] .m.m.a.ExceptionHandlerExceptionResolver : Resolved exception caused by Handler execution: java.lang.NullPointerException

iti · August 14, 2019, 12:18am

There is no easy way to reproduce the error. But the code is like following:
ExpirationDateUtil.getAggregationBuilders(entry.getValue().getPath()).forEach(edrSubAggBuilder -> {

builder.subAggregation(edrSubAggBuilder
         .subAggregation(AggregationBuilders.cardinality("unique_summary_count")
                    .field("relation#summary")));
            });

Before adding is like this and it works fine:
ExpirationDateUtil.getAggregationBuilders(entry.getValue().getPath()).forEach(edrSubAggBuilder -> {

    builder.subAggregation(edrSubAggBuilder);

});

javanna · August 15, 2019, 8:46am

In the stacktrace that you posted, I see an index not found exception being returned, and a null pointer exception in your client code. I don't see any class not found.

iti · August 15, 2019, 4:46pm

Sorry, my bad. Did not pay attention to the error. Here is the correct trace:

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/search/aggregations/metrics/cardinality/ParsedCardinality
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:982)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	...
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/search/aggregations/metrics/cardinality/ParsedCardinality
	at org.elasticsearch.client.RestHighLevelClient.lambda$getDefaultNamedXContents$19(RestHighLevelClient.java:2087)
	at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415)
	at org.elasticsearch.common.xcontent.XContentParserUtils.parseTypedKeysObject(XContentParserUtils.java:153)
	at org.elasticsearch.search.aggregations.bucket.range.ParsedRange$ParsedBucket.parseRangeBucketXContent(ParsedRange.java:182)
	at org.elasticsearch.search.aggregations.bucket.range.ParsedDateRange$ParsedBucket.fromXContent(ParsedDateRange.java:71)
	at org.elasticsearch.search.aggregations.bucket.range.ParsedDateRange.lambda$static$0(ParsedDateRange.java:41)
	at org.elasticsearch.search.aggregations.ParsedMultiBucketAggregation.lambda$declareMultiBucketAggregationFields$0(ParsedMultiBucketAggregation.java:75)
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:314)
	at org.elasticsearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:308)
	at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:340)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:168)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:131)
	at org.elasticsearch.search.aggregations.bucket.range.ParsedDateRange.fromXContent(ParsedDateRange.java:46)
	at org.elasticsearch.client.RestHighLevelClient.lambda$getDefaultNamedXContents$55(RestHighLevelClient.java:2124)
	at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415)
	at org.elasticsearch.common.xcontent.XContentParserUtils.parseTypedKeysObject(XContentParserUtils.java:153)
	at org.elasticsearch.search.aggregations.bucket.ParsedSingleBucketAggregation.parseXContent(ParsedSingleBucketAggregation.java:86)
	at org.elasticsearch.search.aggregations.bucket.filter.ParsedFilter.fromXContent(ParsedFilter.java:34)
	at org.elasticsearch.client.RestHighLevelClient.lambda$getDefaultNamedXContents$51(RestHighLevelClient.java:2120)
	at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415)
	at org.elasticsearch.common.xcontent.XContentParserUtils.parseTypedKeysObject(XContentParserUtils.java:153)
	at org.elasticsearch.search.aggregations.Aggregations.fromXContent(Aggregations.java:141)
	at org.elasticsearch.action.search.SearchResponse.innerFromXContent(SearchResponse.java:292)
	at org.elasticsearch.action.search.SearchResponse.fromXContent(SearchResponse.java:251)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2053)
	at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$13(RestHighLevelClient.java:1697)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1781)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092)
	at xxx.dao.impl.QueryService.buckets(QueryService.java:425)
	at xxx.web.rest.SearchResource.getBuckets(SearchResource.java:95)
	at xxx.web.rest.SearchResource$$FastClassBySpringCGLIB$$cec506a1.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	...
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	... 88 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.search.aggregations.metrics.cardinality.ParsedCardinality
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 142 common frames omitted

2019-08-15 09:37:47.420  WARN 472 --- [  XNIO-2 task-1] .m.m.a.ExceptionHandlerExceptionResolver : Resolved exception caused by Handler execution: org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/search/aggregations/metrics/cardinality/ParsedCardinality

javanna · August 16, 2019, 12:53pm

I cannot think of a reason that would cause besides some classpath issues in your application. The version you are using was compiled correctly and relies on the ParsedCardinality class, but at runtime such class is not there when it needs to be loaded. Given that the high-level client depends on the whole elasticsearch server, you possibly have a version of the client in your classpath, but a different version of elasticsearch which does not support yet the cardinality aggregation. Could you check that please?

iti · August 16, 2019, 4:26pm

Why high level REST API lib comes with 2.4.6 version of elasticsearch?

All my elastic search lib are 6.x and above. Cardinality aggs should be there. 2.4.6 will not though. But I have no control over it.

Thanks

rjernst · August 16, 2019, 6:18pm

The HLRC 6.8.1 depends on Elasticsearch 6.8.1 jar. Your dependencies are overriding that somewhere, that is why it says elasticsearch: 2.4.6 (managed from 6.8.1).

Also, please don't post images of text as they are hardly readable and not searchable. Instead paste the text and format it with </> icon. Check the preview window.

Your application's pom file would be useful to debug this issue.

iti · August 17, 2019, 9:59pm

I am using Spring Boot Starter 1.5.1. It bundles with elasticsearch 2.4.6 jar, that is why 2.4.6 is the managed version. But HLRC 6.8.1 is not smart enough to use the overiding jar. (Strange though, if I lower the Elasticsearch jar to 6.8 instead of 7.x. 6.8.1 HLRC does know to use the 6.8 jar, there is no cardinality error.) So, what is the problem? Latest Spring Boot 2.1.7 is using Elasticsearch 6.4. So, if I upgrade, cardinality should be there. But some other error will happened.
So, I still want to know why overriding jar did not get pick up by 6.8 HLRC correctly when ES jar is 7.x (use 6.8.x ES jar and 6.8.x HLRC jar does not have cardinality error. But when connecting to 7.x ES, some of the aggregation result on 7.X ES is not correct.)

BTW, the image I posted is not a text, it is a window print screen(It is not a text screen). I have no way of formatting it.

dadoonet · August 18, 2019, 4:23am

When using springboot with
elasticsearch, you need to be explicit with some transitive dependencies as SpringBoot declares a version 6.4...

Basically you can put this in your pom.xml:

<properties>
  <elasticsearch.version>7.3.0<elasticsearch.version>
</properties>

See documentation here: https://docs.spring.io/spring-boot/docs/current/reference/html/howto-build.html#howto-customize-dependency-versions

iti · August 18, 2019, 8:59am

Tried that, but the error still the same. I search online and the forum, spring boot and elasticsearch with NoClassDefFoundError so far no solution.

Adding the property just make the managed version to be 7.x. But the error remains.

iti · August 18, 2019, 11:34pm

@javanna @rjernst @dadoonet

I think I have figured out the NoClassDefFoundError problem.

In ES 7.x Lib, there is:
     import org.elasticsearch.search.aggregations.metrics.CardinalityAggregationBuilder;
     import org.elasticsearch.search.aggregations.metrics.ParsedCardinality;

In ES 6.x lib, it is:
     import org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregationBuilder;
     import org.elasticsearch.search.aggregations.metrics.cardinality.ParsedCardinality;

So, when using 6.x HLRC, it is expecting   
    org.elasticsearch.search.aggregations.metrics.cardinality.ParsedCardinality.
But I am using ES 7.x lib which does not have this class, but the
    org.elasticsearch.search.aggregations.metrics.ParsedCardinality

Is there any reason for the name change from 6.x to 7.x? I did not see it in the breaking changes note.
I have no easy way of getting around this problem. Lower ES lib to 6.x will have totalHit() as 0 when accessing ES 7.x, since 7.x change the totalHit() to an object with value and relation, 6.x just a long value.
Anyway Elastic.co can change HLRC 6.x lib or 7.x ES lib to make HLRC 6.x compatible with 7.x ES?
Thanks

javanna · August 22, 2019, 11:11am

It is true that the classes changed package in 7.x, but I don't think that is a breaking change. If you use the 6.x client, it depends on elasticsearch 6.x which has the classes in the right place. If you use the 7.x client, it depends on elasticsearch 7.x which has the classes in the new location, still working though.

I suspect that you are using two different versions of client/server libraries.

We said that if you send requests to both 6.x and 7.x, you should keep on using the 6.x client, which depends on 6.x libraries. That should work.

iti · August 22, 2019, 1:05pm

@javanna Thanks for the reply.
I think in my previous reply, I said that " Lower ES lib to 6.x will have totalHit() as 0 when accessing ES 7.x, since 7.x change the totalHit() to an object with value and relation, 6.x just a long value."
So, using both 6.x client and lib does not work. I think this one you can test it out easily.

javanna · August 22, 2019, 1:30pm

I see what you mean, the point is that we would never recommend to manually change the libs you use. 6.x works with 6.x, it will give unexpected results when used with 7.x libraries, like you are discovering.

When you have problems with total hits, you are hitting this bug: https://github.com/elastic/elasticsearch/issues/43925 . I would suggest to monitor that issue to see when progress is made on it. Meanwhile, I am afraid you will need to use the 7.x client to talk to 7.x nodes, and the 6.x client to talk to 6.x nodes. IN fact, this is the recommended approach and mixed clusters are really only needed while you are upgrading, which is a temporary situation.

iti · August 22, 2019, 5:04pm

Thanks. Looks like I just have to wait for 6.8.x HLRC for the fix.

iti · September 12, 2019, 9:11pm

With 6.8.3 HLRC and 6.8.3 ES lib. I was able to access 6.x and 7.x ES with correct TotalHit() and no Cardinality class not found error.

Thanks for all the help.

system · October 10, 2019, 9:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ElasticSearch High Level Client ClassNotFoundError Elasticsearch	2	1307	December 13, 2017
Error while using Java High Level Rest Client Elasticsearch	8	14072	April 17, 2018
Class not found exception for org.apache.lucene.util.Version Elasticsearch	12	4148	November 24, 2018
java.lang.ClassNotFoundException: org.elasticsearch.common.CharArrays Elasticsearch	1	1516	June 5, 2019
Ccs_minimize_roundtrips error when using 7.X high level REST API to access 6.x index Elasticsearch	3	7000	September 6, 2019

java.lang.ClassNotFoundException: org.elasticsearch.search.aggregations.metrics.cardinality.ParsedCardinality

Related topics