Elasticsearch Healthcheck failure w/ Spring Boot

[apizano]

I posted on an issue on the spring boot GitHub, and was referred here. I will try to make this is simple as possible so I can get some help. Its actually quite simple.

TL;DR

• Used Spring Boot actuator to create custom Healthcheck monitor for Elasticsearch service

• Created 1 custom Java class called "IndexExists" (code below)

• Added the Application.yml file: rest.uri = ['our-dev-url-on-aws'] property

I have an application that works perfectly fine both locally and remotely, however when adding a custom Healthcheck monitor using Spring Boot to monitor my Elasticsearch service, I get a "connection refused" to Elasticsearch and the health check monitor ultimately fails. Our Load Balancer on AWS attempts to hit this endpoint and because it returns a status: "DOWN" because the healthcheck cannot connect to Elasticsearch, the Load Balancer starts to create a new Container. When looking at the CloudWatch logs, this happens repeatedly in an infinite loop (the Load Balancer attempting to make more containers). I would like to add, this works perfectly fine locally - that is, when adding the healthcheck monitor, and using a HTTP GET request on the actuator/health endpoint via POSTMAN, I get the correct JSON response which is:

{
    "status": "UP",
    "details": {
        "indexExists": {
            "status": "UP",
            "details": {
                "index": "exists",
                "value": "assets"
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 250790436864,
                "free": 194987540480,
                "threshold": 10485760
            }
        },
        "elasticsearchRest": {
            "status": "UP",
            "details": {
                "cluster_name": "elasticsearch",
                "status": "yellow",
                "timed_out": false,
                "number_of_nodes": 1,
                "number_of_data_nodes": 1,
                "active_primary_shards": 7,
                "active_shards": 7,
                "relocating_shards": 0,
                "initializing_shards": 0,
                "unassigned_shards": 5,
                "delayed_unassigned_shards": 0,
                "number_of_pending_tasks": 0,
                "number_of_in_flight_fetch": 0,
                "task_max_waiting_in_queue_millis": 0,
                "active_shards_percent_as_number": 58.333333333333336
            }
        }
    }
}

As you can see, the Healthcheck monitor is returning the "details": { "indexExists": etc... } portion up at the top, which checks if my Elasticsearch index is mapped to the string = "assets". If it is, then it returns "status": "UP".

However, when pushing this code to the build pipeline in Azure, so I can test in a dev environment, this is the JSON response I get:

{
    "status": "DOWN",
    "details": {
        "indexExists": {
            "status": "UP",
            "details": {
                "index": "exists",
                "value": "assets"
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 16776032256,
                "free": 9712218112,
                "threshold": 10485760
            }
        },
        "elasticsearchRest": {
            "status": "DOWN",
            "details": {
               "error": "java.net.ConnectException: Connection refused"
            }
        }
    }
}

I can view the error logs on our AWS (Amazon Web Services) cluster and they look like this:

The java class that I created and added to our code base is "IndexExists.java" and it is here:

package com.cat.digital.globalsearch.component;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;
import com.cat.digital.globalsearch.data.Indices;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;


@Component
public class IndexExists implements HealthIndicator {
    private final ESClient esClient;
    private static final Logger LOGGER = LoggerFactory.getLogger(IndexExists.class);
    Map<String,String> map = new HashMap<>();


    @Autowired
    public IndexExists(ESClient esClient) {
        this.esClient = esClient;
    }

    @Override
    public Health health() {
        try {
            if (!esClient.hasIndex(Indices.INDEX_ASSETS)) {
                return Health.down().withDetail("index", Indices.INDEX_ASSETS + " index does not exist").build();
            }
        } catch (IOException e) {
            LOGGER.error("Error checking if Elasticsearch index {} exists , with exception", Indices.INDEX_ASSETS,  e);
        }
        map.put("index","exists");
        map.put("value", Indices.INDEX_ASSETS);
        return Health.up().withDetails(map).build();
    }
}

I won't post all the code of the application.yml, but here are the bits I added. For the spring dev profile, I only added the rest uri, the rest of the code was already there:

management:
  endpoint:
    health:
      show-details: always

spring:
  profiles: dev
  data:
    elasticSearch:
    host: "was-dev-url"
    port: -1
    scheme: https
    rest:
    uris: ["https://aws-dev-url"]

I hope that information isn't too much! I really need help... If anyone needs any more information, please let me know. Thank you.

[apizano]

Well, I found a stack overflow post that could relate to my issue:

Springboot elastic search health management : ConnectException: Connection refused

The config I am using for ESClient that I Autowire in the "IndexExists.java" class looks like this:
package com.cat.digital.globalsearch.configuration;

import com.cat.digital.globalsearch.component.ESClient;
import org.apache.http.HttpHost;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ESClientConfig {

@Value("${elasticSearch.host}")
private String esHost;

@Value("${elasticSearch.port}")
private int esPort;

@Value("${elasticSearch.scheme}")
private String esScheme;

@Bean
public ESClient esClient() {
return new ESClient( new HttpHost(esHost, esPort, esScheme));
}
}

and this is the initial part of the class and the constructor:

public class ESClient {

private final RestClientBuilder builder;
private final RestHighLevelClient searchClient;

public ESClient(HttpHost... hosts) {
    this.builder = RestClient.builder(hosts);
    searchClient = new RestHighLevelClient(RestClient.builder(hosts));
}

}

so if the if those variables and such are used for the RestHighLevelClient for the Elasticsearch mapping, indexing, and etc.... but Elasticsearch is using the Rest client for the Healthcheck actuator, that could explain the failure?

[apizano]

Looks like I found a solution, refer here:

2.1.6 Spring Boot - Elasticsearch Healthcheck failure