Java API Client Conflict? java.lang.NoSuchMethodError: org.elasticsearch.client.RequestOptions$Builder.setHttpAsyncResponseConsumerFactory

[linusField]

Hi!

We are trying to upgrade from Elastic Search 7 to 8. We have been using the Rest High Level Client 7.17.22 and want to upgrade it to 8.14.3.

As suggested by the migration guide, we should be able to have both versions of the clients running together:

Migrating from the High Level Rest Client | Elasticsearch Java API Client...

However, when we try to initialize the version 8 RestApiClient, we are getting the following exception

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class co.elastic.clients.transport.rest_client.SafeResponseConsumer
	org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1087)
	org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
	org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
	org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
	net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
	org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:186)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	com.arkplatforms.arkit.web.servlet.filter.CacheHeaderFilter.doFilter(CacheHeaderFilter.java:49)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	com.arkplatforms.arkit.web.servlet.filter.JsonpCallbackFilter.doFilter(JsonpCallbackFilter.java:69)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	com.arkplatforms.arkit.web.servlet.filter.CustomSecurityRedirectFilter.doFilter(CustomSecurityRedirectFilter.java:44)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352)
	org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117)
	org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
	org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:168)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
	org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:151)
	org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:129)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
	org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
	org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
	org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)

We were also able to dig up an old issue around this, which seems to have to do with library conflict issues between 7 and 8, so not sure if it is related:

ElasticSearch 8.12.1 with HighLevelRestClient/JavaClient combination not working

Here is our pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>parent-pom</artifactId>
        <groupId>com.mycompany</groupId>
        <version>1.0</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mycompany</groupId>
    <artifactId>search</artifactId>
    <packaging>jar</packaging>
    <version>3.0.0-SNAPSHOT</version>

    <dependencies>
        <!-- Arkplatforms commons -->
        <dependency>
            <groupId>com.mycompany</groupId>
            <artifactId>commons</artifactId>
            <version>3.0.0-SNAPSHOT</version>
            <scope>compile</scope>
        </dependency>

        <!-- Elasticsearch Java Client -->
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.17.22</version>
        </dependency>

        <dependency>
            <groupId>co.elastic.clients</groupId>
            <artifactId>elasticsearch-java</artifactId>
            <version>8.14.3</version>
        </dependency>

        <!--elasticsearch stuff for geo-queries -->
        <!--see https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-geo-queries.html-->

        <dependency>
            <groupId>org.locationtech.spatial4j</groupId>
            <artifactId>spatial4j</artifactId>
            <version>0.7</version>
        </dependency>

        <dependency>
            <groupId>org.locationtech.jts</groupId>
            <artifactId>jts-core</artifactId>
            <version>1.15.0</version>
            <exclusions>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xercesImpl</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

    </dependencies>
</project>

And here is the client code:

private ElasticsearchClient initializeRestClient() {
        
        RestClientBuilder builder = RestClient.builder(new HttpHost(url, port, protocol));

        if (StringUtils.isNotEmpty(username)) {
            final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
            credentialsProvider.setCredentials(AuthScope.ANY,
                    new UsernamePasswordCredentials(username, password));

            builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
        }
        RestClient restClient = builder.build();

        ElasticsearchTransport transport = new RestClientTransport(
                restClient, new JacksonJsonpMapper());

        // And create the API client
        return new ElasticsearchClient(transport);
    }

    @Override
    public Boolean indexExists(String indexName) {
        Boolean result;
        String actualIndexName = arkitTarget + "_" + indexName;

        ElasticsearchClient restClient = initializeRestClient();

        try {
            BooleanResponse exists = restClient.indices().exists(ExistsRequest.of(e -> e.index(indexName)));
            result = exists.value();
        } catch (Exception e) {
            LOG.error("Error when initializing search index " + actualIndexName, e);
            return null;
        } finally {
            closeClient(restClient);
        }

        return result;
    }


    private void closeClient(final ElasticsearchClient restClient) {
        try {
            restClient._transport().close();
        } catch (Exception e) {
            LOG.error("ElasticSearch client did not close properly", e);
        }
    }

Is there something that we are missing or is there some way that we are not initializing the client correctly?

Thanks.

[ltrotta (Laura Trotta)]

Hello!
I have created a minimal project using the pom and code provided, and I didn't manage to reproduce the error - with only the two elasticsearch dependencies + jackson + the spring bom everything works fine.

SafeResponseConsumer is a class in found in version 8 of the client which extends HttpAsyncResponseConsumer from apache httpcore-nio, I suspect there could be some dependency clashing with other imported libraries (spring security for example) that resolves to the project using a non compatible version of the apache library.

I suggest following the guide provided in the old thread, because the problem seems to be similar.