Cannot get specific prefix index names with ElasticSearch

Eric_Chen · January 17, 2024, 3:11am

Hi everyone, I've run into an issue with Elasticsearch on our company's internal network and I'm not sure how to go about troubleshooting it. I need to extract information about indices (not documents) from my ES that have a specific prefix. However, I'm encountering a 'not found' error when I enter this prefix, despite the fact that these indices do exist in my ES. I haven't been able to replicate this problem in my local (internet-connected) environment, so setting up a test scenario is challenging. I realize getting a precise answer might be difficult, but I'd appreciate any troubleshooting tips or key points you could provide to help me better understand what might be causing this issue.
env:

elasticseach java client : 7.17.14
jdk:17
elasticsearch server: 8.8.2
my code:


import org.apache.http.HttpHost;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.ssl.SSLContextBuilder;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.RestHighLevelClientBuilder;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.client.indices.GetIndexResponse;
import org.junit.Test;

import javax.net.ssl.SSLContext;
import java.io.IOException;
import java.net.URI;
import java.util.Arrays;

public class QuickElasticSearchClientTest {
    @Test
    public void test_() throws Exception {
        URI uri = new URI("https://my.internal.domain.com");
        String scheme = uri.getScheme();
        String host = uri.getHost();
        int port = uri.getPort() != -1 ? uri.getPort() : (scheme.equals("https") ? 443 : 80);
        SSLContext sslContext = new SSLContextBuilder()
                .loadTrustMaterial(null, (chain, authType) -> true).build();
        RestClient httpClient = RestClient.builder(
                new HttpHost(host, port, scheme)
        ).setHttpClientConfigCallback(
                httpAsyncClientBuilder -> httpAsyncClientBuilder.setSSLContext(sslContext).setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
        ).build();
        RestHighLevelClient client = new RestHighLevelClientBuilder(httpClient).setApiCompatibilityMode(true).build();
        GetIndexRequest getIndexRequest = new GetIndexRequest("tasklist*");
        GetIndexResponse response = null;
        try {
            response = client.indices().get(getIndexRequest, RequestOptions.DEFAULT);
            Arrays.stream(response.getIndices()).forEach(System.out::println);
        } catch (IOException | RuntimeException e) {
            System.out.println(String.format("%s - cause: %s",e.getMessage(),e.getCause()));
        }
        Arrays.stream(client.indices().get(new GetIndexRequest("tasklis*"), RequestOptions.DEFAULT).getIndices()).forEach(
                System.out::println
        );
    }


}

response :

Unable to parse response body - cause: org.elasticsearch.client.ResponseException: method [GET], host [https://my.internal.domain.com:443], URI [/tasklist*?master_timeout=30s], status line [HTTP/1.1 404 Not Found]
<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1></body></html>
tasklist-draft-task-variable-8.3.0_
tasklist-flownode-instance-8.3.0_
tasklist-form-8.3.0_
tasklist-import-position-8.2.0_
tasklist-metric-8.3.0_
tasklist-migration-steps-repository-1.1.0_
tasklist-process-8.3.0_
tasklist-process-instance-8.3.0_
tasklist-task-8.3.0_
tasklist-task-variable-8.3.0_
tasklist-user-1.4.0_
tasklist-variable-8.3.0_
tasklist-web-session-1.1.0_

ElasticsearchStatusException[Unable to parse response body
]; nested: ResponseException[method [GET], host [https://my.internal.domain.com:443], URI [/tasklist*?master_timeout=30s], status line [HTTP/1.1 404 Not Found]
<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1></body></html>];
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2464)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118)
	at org.elasticsearch.client.IndicesClient.get(IndicesClient.java:1067)
	at com.wxt.fwx.flow.camunda.zeebe.tasklist.domain.elastic.QuickElasticSearchClientTest.test_(QuickElasticSearchClientTest.java:40)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
	Suppressed: java.lang.IllegalStateException: Unsupported Content-Type: text/html;charset=utf-8
		at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2481)
		at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461)
		... 32 more
Caused by: org.elasticsearch.client.ResponseException: method [GET], host [https://my.internal.domain.com:443], URI [/tasklist*?master_timeout=30s], status line [HTTP/1.1 404 Not Found]
<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1></body></html>
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)

http client to quest:


public class Main {
    public static void main(String[] args) throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
        SSLContext sslContext = new SSLContextBuilder()
                .loadTrustMaterial(null, (chain, authType) -> true).build();
        CloseableHttpClient httpClient = HttpClientBuilder.create().setSSLContext(sslContext).setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE).build();

        try {
            // 创建GET请求
            HttpGet request = new HttpGet("https://my.internal.domain.com/_cat/indices/tasklist*");

            // 执行请求并获取响应
            HttpResponse response = (HttpResponse) httpClient.execute(request);

            // 打印响应状态码
            System.out.println("Response status code: " + response.getStatusLine().getStatusCode());

            // 打印响应内容
            String responseBody = EntityUtils.toString(response.getEntity());
            System.out.println("Response body: " + responseBody);
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                httpClient.close();
            } catch (Exception e) {
                e.printStackTrace();
            }

        }
    }
}

Response status code: 200
Response body: green open tasklist-migration-steps-repository-1.1.0_ R8QVrzPLRySIgfrZ7xtupQ 1 0 22 2 71.3kb 71.3kb
green open tasklist-process-8.3.0_                    _0vc5Fx8RGeSxcn4cYJ0CQ 1 0  1 0  6.9kb  6.9kb
green open tasklist-web-session-1.1.0_                fgNb0MVAR52atmCLz-E5uA 1 0  0 0   247b   247b
green open tasklist-user-1.4.0_                       vE380dVeQdyZhz5yDQU8-w 1 0  0 0   247b   247b
green open tasklist-metric-8.3.0_                     R1_-ITtNQSqIjn7vYLRgIQ 1 0  0 0   247b   247b
green open tasklist-variable-8.3.0_                   lzfP-RJgQe6U88SNrwLHRQ 1 0  0 0   247b   247b
green open tasklist-import-position-8.2.0_            RDoK1sNBQOGb7wFJ_uTyHQ 1 0  1 0  5.9kb  5.9kb
green open tasklist-form-8.3.0_                       HNmSPn7CTNu8iXM98uS1DA 1 0  0 0   247b   247b
green open tasklist-task-variable-8.3.0_              gXMuEyUTT52l7Lsh-sq4aA 1 0  0 0   247b   247b
green open tasklist-draft-task-variable-8.3.0_        wpvtrJsEQDSXQcoGxd-RAw 1 0  0 0   247b   247b
green open tasklist-flownode-instance-8.3.0_          7VS0JAUjT1SLV4papQ8v7w 1 0  0 0   247b   247b
green open tasklist-task-8.3.0_                       WFev12-rQVmYK6GCfp5bLQ 1 0  0 0   247b   247b
green open tasklist-process-instance-8.3.0_           WTfAbjWNQ7uklpNTgD9wAQ 1 0  0 0   247b   247b

If I can provide more information, please let me know.

dadoonet · January 17, 2024, 3:27am

Welcome!

Do you have a proxy or something? The response (HTML) is not from Elasticsearch.

Any reason why you are not using the same version for the client as for the server? Use the new Java client instead.

Eric_Chen · January 17, 2024, 3:39am

Hi there,
thanks for your reply！
I wanted to share some details about our Elasticsearch setup and the issue we're encountering. Our Elasticsearch is running in a Kubernetes cluster within our company's internal network, and I access the Elasticsearch service via HTTPS through the cluster's gateway. We're currently using Elasticsearch 7.17.14 and RestHighLevelClient 7.17.14 and would prefer to avoid making major changes.

Interestingly, in my own development environment, which has internet access, I've found that using the same versions (Elasticsearch server and Java client) allows me to retrieve data without any problems. Also, if I use any prefix shorter than 'tasklist' in my queries, as per my code, it works just fine. However, as soon as I try something longer, like 'tasklist-', it stops working.

Looking forward to your insights or suggestions on this matter.

Best regards,
eric

Eric_Chen · January 17, 2024, 3:42am

Sorry, my English isn't very good. I might have misunderstood your meaning. Should I try to directly connect to my ES server?

dadoonet · January 17, 2024, 11:18am

Is it using the ECK operator?

May be there's a misconfiguration on that side?
I'm not a K8s user so I can't really help.

I don't think it's related at all. But the fact that you are running locally is a major change

May be that's a problem again with the proxy.

Eric_Chen · January 19, 2024, 12:58am

That might be the reason, and if I later figure out the exact cause, I will provide feedback here.

Eric_Chen · January 22, 2024, 6:23am

I believe I have found the reason. Our Kubernetes (k8s) has set a special permission for anything starting with "tasklist", so when access requires permission verification, that's where my problem lies.

system · February 19, 2024, 6:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.