Querying elasticsearch during a rolling update

Lucy_Johnson · March 26, 2024, 8:26am

Hello everyone. I'm not super sure if that's the best place to ask but I'm hoping to get further information from here.

My goal is to query my elasticsearch-cluster (v 7.17) with either the deprecated elastic client or the recommended one (Java) during a rolling update.

To enable that I'm using the helm chart from bitnami charts/bitnami/elasticsearch at main · bitnami/charts · GitHub that abstracts all the rolling update mechanisms for Kubernetes away, so I don't have to do that myself as described in the rolling update tutorial (Rolling upgrades | Elasticsearch Guide [7.17] | Elastic).

I've passed my elastic-client the addresses to my coordinator nodes (coordinator-0 and coordinator-1). And tried to perform a rolling update while doing some simple queries in a test Java-App in a loop. Unfortunately I'm receiving an "UnknownHostException" error once the coordinator-0 shuts down. (That should be normal, since the bitnami chart uses the Kubernetes stateful set, that doesn't deliver a DNS lookup anymore once the node is down)

I would have expected the client to then try to perform the request on coordinator-1 next, but it exits with the mentioned Exception (for both Java Elastic-Clients) instead. One reason for that seems to be that the UnknownHostException is not a RetryableException as required to perform the Request on the next node:

github.com

elastic/elasticsearch/blob/main/client/rest/src/main/java/org/elasticsearch/client/RestClient.java#L306


      
          throws IOException {
          RequestContext context = request.createContextForNextAttempt(tuple.nodes.next(), tuple.authCache);
          HttpResponse httpResponse;
          try {
              httpResponse = client.execute(context.requestProducer, context.asyncResponseConsumer, context.context, null).get();
          } catch (Exception e) {
              RequestLogger.logFailedRequest(logger, request.httpRequest, context.node, e);
              onFailure(context.node);
              Exception cause = extractAndWrapCause(e);
              addSuppressedException(previousException, cause);
              if (isRetryableException(e) && tuple.nodes.hasNext()) {
                  return performRequest(tuple, request, cause);
              }
              if (cause instanceof IOException) {
                  throw (IOException) cause;
              }
              if (cause instanceof RuntimeException) {
                  throw (RuntimeException) cause;
              }
              throw new IllegalStateException("unexpected exception type: must be either RuntimeException or IOException", cause);
          }

So I'm in a difficult position now, where I'm not sure how to proceed further. Should I implement my own retry-mechanism in the Application, or am I using the chart wrong in some way?

I hoped that my use case would be fairly common, so that I could just use the defaults of the bitnami chart. if you guys have some ideas I would be happy for any kind of advice.

Cheers

system · April 23, 2024, 8:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.