Dear All,
My new ECE cluster is up and running however I am getting below message when I access kibana portal directly on port serverIP:9243. I am able to access the admin console of the cluster but no the kibana.
{"ok":false,"message":"Unknown deployment."}
Not sure what is wrong. Any help will be appreciated. Right now I am trying to access one server kibana directly before moving to start using LB/haproxy
Waiting for some pointers from experts to resolve my problem.
I also tried with haproxy but still not working. Another thing noticed that http requests are forwarded through haproxy but not the https requests again not sure why and still get the same error
{"ok":false,"message":"Unknown deployment."}
Below is my sample haproxy config
My ECE cluster is running on a private cloud and have 3 allocator hosts and haproxy is configured on first allocator host (192.168.1.1).
It has 3 master, 3 kibana and 3 data nodes.
My platform->settings has entry of "192.1681.1.1.ip.es.io"
Not sure what is wrong. Any help will be appreciated
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
defaults
mode http
log global
option httplog
option dontlognull
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen ece_coordinator_uiapi_http_12400
bind 192.168.1.1:1240
# bind *:12400
#mode tcp
mode http
balance source
#option tcplog
reqadd X-Forwarded-Proto:\ http
server allocator01_12400 192.168.1.1:12400 check
server allocator02_12400 192.168.1.1:12400 check
server allocator03_12400 192.168.1.1:12400 check
listen ece_coordinator_uiapi_https_12443
bind 192.168.1.1:2443
#bind *:12443
#mode tcp
mode http
balance source
#option tcplog
reqadd X-Forwarded-Proto:\ https
server allocator01_12443 192.168.1.1:12443 check
server allocator02_12443 192.168.1.2:12443 check
server allocator03_12443 192.168.1.3:12443 check
listen ece_proxy_elastickibana_http_9200
bind 192.168.1.1:9100
#bind *:9200
#mode tcp
mode http
balance source
#option tcplog
reqadd X-Forwarded-Proto:\ http
server allocator01_9200 192.168.1.1:9200 check
server allocator02_9200 192.168.1.2:9200 check
server allocator03_9200 192.168.1.3:9200 check
listen ece_proxy_elastickibana_https_9243
bind 192.168.1.1:9223
#bind *:9243
#mode tcp
mode http
balance source
#option tcplog
reqadd X-Forwarded-Proto:\ https
server allocator01_9243 192.168.1.1:9243 check
server allocator02_9243 192.168.1.2:9243 check
server allocator03_9243 192.168.1.3:9243 check
listen stats
bind 192.168.1.1:9090
#bind *:9090
mode http
stats uri /
stats auth user1:password
Either the hostname needs to start with the "cluster id" of the ES or Kibana (the UI has links to the right URLs for each deployment), or you need to inject the header "X-found-cluster: <<id>>" (if using haproxy/nginx)
There's some documentation kicking around, eg: https://www.elastic.co/guide/en/cloud-enterprise/current/ece-administering-endpoints.html
Hi Alex,
This time got a 503 error post your suggested changes
[elastic@ecehost1 haproxy]$ curl -k -u elastic http://192.168.1.1:9223
Enter host password for user 'elastic':
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
[elastic@ecehost1 haproxy]$
[elastic@ecehost1 haproxy]$ curl -k -u elastic http://72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9223
Enter host password for user 'elastic':
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
[elastic@ecehost1 haproxy]$
[elastic@ecehost1 haproxy]$ curl -u elastic http://72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9223
Enter host password for user 'elastic':
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
[elastic@ecehost1 haproxy]$
listen ece_proxy_elastickibana_https_9243
bind 192.168.1.1:9223
#bind *:9243
#mode tcp
mode http
balance source
#option tcplog
http-request add-header X-Found-Cluster 72bbc3de0e144f378ebab582fe3ea047
reqadd X-Forwarded-Proto:\ https
server allocator01_9243 192.168.1.1:9243 check
server allocator02_9243 192.168.1.2:9243 check
server allocator03_9243 192.168.1.3:9243 check
I would get curl working directly to 9243 first, then it just becomes an haproxy config problem (that 503 comes from haproxy so my guess is none of the allocators show as available via whatever "check" does, you can probably coax haproxy into spitting out some useful logs)
hhmm I tried even with 9243 directly by skipping haproxy and gives me same error. Even curl over 9200 is giving same error
Any way to check the status of kibana from command line? Though UI it shows in Healthy state
[elastic@ecehost1 haproxy]$ curl -k -u elastic https://72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9243
Enter host password for user 'elastic':
curl: (56) Received HTTP code 503 from proxy after CONNECT
[elastic@ecehost1 haproxy]$ curl -k -u elastic https://192.168.1.1:9243
Enter host password for user 'elastic':
curl: (56) Received HTTP code 503 from proxy after CONNECT
[elastic@ecehost1 haproxy]$ curl -u elastic https://72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9243
Enter host password for user 'elastic':
curl: (56) Received HTTP code 503 from proxy after CONNECT
[elastic@ecehost1 haproxy]$ curl -u elastic https://192.168.1.1:9243
Enter host password for user 'elastic':
curl: (56) Received HTTP code 503 from proxy after CONNECT
[elastic@ecehost1 haproxy]$
ok my bad. I forgot to remove proxy from elastic env during testing. Post removing proxy I get status as healthy but can't access the kibana portal
[elastic@ecehost1 ~]$ curl -k -u elastic https://192.168.1.1:9243
Enter host password for user 'elastic':
{"ok":false,"message":"Unknown deployment."}
[elastic@ecehost1 ~]$ curl -k -u elastic https://192.168.1.1:9243/_health
Enter host password for user 'elastic':
{
"ok": true,
"status": 200
}
[elastic@ecehost1 ~]$ curl -u elastic https://192.168.1.1:9243
Enter host password for user 'elastic':
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
[elastic@ecehost1 ~]$
Health status log below:
[elastic@ecehost1 ~]$ curl -k -u elastic https://72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9243/_health
Enter host password for user 'elastic':
{
"ok": true,
"status": 200
}
[elastic@ecehost1 ~]$
---------------------------
{"type":"response","@timestamp":"2020-08-10T16:29:13Z","tags":["api"],"pid":32,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"192.168.1.1:18350","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"172.17.42.1","userAgent":"172.17.42.1"},"res":{"statusCode":200,"responseTime":42,"contentLength":9},"message":"GET /api/status 200 42ms - 9.0B"}
I also tried to hit a non-existent page and checked the logs, below are the output
[elastic@ecehost1 ~]$ curl -k -u elastic https://72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9243/apps
Enter host password for user 'elastic':
{"statusCode":404,"error":"Not Found","message":"Not Found"}
[elastic@ecehost1 ~]$
Below is the snippet log of above command
{"type":"response","@timestamp":"2020-08-10T16:30:38Z","tags":[],"pid":32,"method":"get","statusCode":404,"req":{"url":"/apps","method":"get","headers":{"host":"192.168.1.1:18350","user-agent":"curl/7.29.0","accept":"*/*","x-cloud-request-id":"vSsroRS-SMGkcI-R8dJ9jw","x-forwarded-host":"72bbc3de0e144f378ebab582fe3ea047.192.168.1.1:9243","x-found-kibana-cluster":"72bbc3de0e144f378ebab582fe3ea047"},"remoteAddress":"172.17.42.1","userAgent":"172.17.42.1"},"res":{"statusCode":404,"responseTime":300,"contentLength":9},"message":"GET /apps 404 300ms - 9.0B"}
This means its hitting the kibana instance but not sure why the page is not opening up
Dear experts,
Is there any solution for this issue?
Appreciate your help here
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.