Hello everyone,
I have strange problem with _cat/indices on a client deployment:
- it always fails on the client cluster,
- it always works using curl with the same parameters,
- I never reproduced it on a test cluster of the same version.
Do you have any idea of what could happen ?
Context
Zeenea is a connected data catalog. It owns among others an Elasticsearch Connector that extract the list of the indexes and their mapping and therefore allow people easily write documentation about there Elasticsearch indices.
The connector uses Java Low Level REST Client (7.1.1) and supports clusters version from 5 to 7 with or without Security enabled.
What I want to do
I do two queries :
-
GET /— to get the major version number so I can know how to deal with types. -
GET _cat/indices— to discover the indices.
My problem
I have a client, with an 6.3.1 ECE cluster, with X-Pack Security enabled for which the second request fails with an unexpected message (code 400): "Illegal request-target, unexpected character '/' at position 4"
When I try the same request from the same machine with curl, it works and I get the expected result.
Traces
Here are the traces. (I changed some server names and secrets.)
1 - Application logs
GET / is 200 OK
2019-09-26 16:48:18.311 INFO zeenea-task-pool-0 zeenea.connector.elasticsearch.client.ElasticsearchClient - connect to elasticsearch cluster 5d9e...86.ece.example.com:9243
2019-09-26 16:48:18.818 DEBUG zeenea-task-pool-0 org.elasticsearch.client.RestClient - request [GET https://5d9e...86.ece.example.com:9243/] returned [HTTP/1.1 200 OK]
2019-09-26 16:48:18.819 TRACE zeenea-task-pool-0 tracer - curl -iX GET 'https://5d9e...86.ece.example.com:9243/'
# HTTP/1.1 200 OK
# Server: fp/fd8165
# Date: Thu, 26 Sep 2019 14:48:18 GMT
# X-Found-Handling-Cluster: 5d9e...86
# X-Found-Handling-Instance: instance-0000000051
# X-Found-Handling-Server: 10.0.0.129
# Content-Type: application/json; charset=UTF-8
# Content-Length: 524
#
# {
# "name" : "instance-0000000051",
# "cluster_name" : "5d9e...86",
# "cluster_uuid" : "something",
# "version" : {
# "number" : "6.3.1",
# "build_flavor" : "default",
# "build_type" : "tar",
# "build_hash" : "eb782d0",
# "build_date" : "2018-06-29T21:59:26.107521Z",
# "build_snapshot" : false,
# "lucene_version" : "7.3.1",
# "minimum_wire_compatibility_version" : "5.6.0",
# "minimum_index_compatibility_version" : "5.0.0"
# },
# "tagline" : "You Know, for Search"
# }
GET /_cat/indices is 400 Bad Request
2019-09-26 16:48:18.846 DEBUG zeenea-task-pool-0 org.elasticsearch.client.RestClient - request [GET https://5d9e...86.ece.example.com:9243/_cat/indices] returned [HTTP/1.1 400 Bad Request]
2019-09-26 16:48:18.846 TRACE zeenea-task-pool-0 tracer - curl -iX GET 'https://5d9e...86.ece.example.com:9243/_cat/indices'
# HTTP/1.1 400 Bad Request
# Server: fp/fd8165
# Date: Thu, 26 Sep 2019 14:48:18 GMT
# Content-Type: text/plain; charset=UTF-8
# Content-Length: 62
#
# Illegal request-target, unexpected character '/' at position 4
2019-09-26 16:48:18.849 ERROR zeenea-task-pool-0 zeenea.connector.spi.DatastorageConnector - Unable to list /
org.elasticsearch.client.ResponseException: method [GET], host [https://5d9e...86.ece.example.com:9243], URI [_cat/indices], status line [HTTP/1.1 400 Bad Request]
Illegal request-target, unexpected character '/' at position 4
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
at zeenea.connector.elasticsearch.client.ElasticsearchClient.$anonfun$performRequest$1(ElasticsearchClient.scala:28)
[... Many more unrelated lines...]
2 - Curl
GET / is 200 OK
$ curl -iX GET 'https://5d9e...86.ece.example.com:9243/' -k --user someone:*******
HTTP/1.1 200 OK
Server: fp/fd8165
Date: Thu, 26 Sep 2019 15:03:28 GMT
X-Found-Handling-Cluster: 5d9e...86
X-Found-Handling-Instance: instance-0000000044
X-Found-Handling-Server: 10.0.0.112
Content-Type: application/json; charset=UTF-8
Content-Length: 524
{
"name" : "instance-0000000044",
"cluster_name" : "5d9e...86",
"cluster_uuid" : "Q5UqxpZuRWGWZY0Gx5flwA",
"version" : {
"number" : "6.3.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "eb782d0",
"build_date" : "2018-06-29T21:59:26.107521Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
GET /_cat/indices is 200 OK
$ curl -iX GET 'https://5d9e...86.ece.example.com:9243/_cat/indices' -k --user someone:*******
HTTP/1.1 200 OK
Server: fp/fd8165
Date: Thu, 26 Sep 2019 15:03:56 GMT
X-Found-Handling-Cluster: 5d9e...86
X-Found-Handling-Instance: instance-0000000051
X-Found-Handling-Server: 10.0.0.129
Content-Type: text/plain; charset=UTF-8
Content-Length: 9798
green open xxxxxxxxxxxx_2019-03-13-21-00-50 U6qxQhOE4BwGhLqefqA3V6 7 1 44892945 9870494 286.4gb 143.1gb
green open xxxxxxxxx 4tHgHypjGxdcUJ4w1OtjFg 1 0 103 6 6mb 6mb
green open xxxxxxxxxxx_2019-09-26-06-42-53 Bw8UiEWPushQHWAEEXVbJi 10 1 7861268592 1106444572 1.1tb 598.5gb
green open xxxxxxxxxxxx_v20190305 Op38IuwyYa6kb9aaek0YSf 1 1 13585 0 11.1mb 5.5mb
[... more items ...]
Thank you,