Kibana “Failed to load indices” and “Failed to load remote clusters”

[ab2obek (Abdel)]

Hi everyone,

I recently installed an Elasticsearch cluster (3 mdi machines), Kibana (1 machine), both on v7.3.0 and everything seems to be working.

My problem is that when I click on "Create Index Pattern" from Kibana's management tab, it notifys "Failed to load indices" and "Failed to load remote clusters" but i can see all indexes on the index pattern form, I can load them and create a pattern.

The failed to load error :

In the "network" tab of page inspection on chrome I can see two requests that goes into a "http 500 error"

The urls are :
https://our_fqdn/kibana/beta/elasticsearch/*/_search?ignore_unavailable=true
https://our_fqdn/kibana/beta/elasticsearch/*%3A*/_search?ignore_unavailable=true

Some infos about my setup

FQDN ** ----> ** httpd** ---->** Kibana / Elastic (port 5601)

The config files :

elasticsearch.yml

network.host: 127.0.0.1, eth2_ip
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ip_of_master1,ip_of_master2,ip_of_master3
indices.fielddata.cache.size: 50%
reindex.remote.whitelist: "master1:9200, master2:9200, master3:9200"

kibana.yml

server.port: "5601"
server.host: "IP_of_the_server"
server.basePath: "/kibana/beta"
server.rewriteBasePath: false
elasticsearch.hosts: "http://localhost:9200"
elasticsearch.username: "user"
elasticsearch.password: "pass"

httpd config

<Location /kibana/beta>
ProxyPass http://kibana_ip:90
ProxyPassReverse http://kibana_ip:90
</Location>

I really can't find what is wrong so that i'm getting this error.
What i've tried :

• I tried to activate the rewrite.Path (negative)
• I tried the to comment reindex.remote.whitelist: "master1:9200, master2:9200, master3:9200" (negative)
• I tried the curl locally and the only curl seems to work is
  curl -XPOST "http://localhost:9200/elasticsearch/*/_search?ignore_unavailable=true"
  So maybe the base request should be
  https://our_fqdn/elasticsearch/*/_search?ignore_unavailable=true without the "kibana/beta" part ?

If anyone got an idea it would be greatly appreciated.

PS: For confidentiality issues I replaced all the IPs with some names like for "master1", "master2" but in my config file it's actually IP addresses

[lukas (Lukas Olson)]

Hmm, are you seeing any errors in the Kibana server console?

You mentioned curl -XPOST "http://localhost:9200/elasticsearch/*/_search?ignore_unavailable=true" is successful, but do you really mean http://localhost:9200/*/_search?ignore_unavailable=true"(without the elasticsearch portion)? If the former is working for you then that is certainly confusing.

[ab2obek (Abdel)]

Hi Lukas,

Actually the curl that worked in the Kibana console is the one sent as a POST when i try to create an index pattern. Here is the result in Kibana console :

There is no HTTP 500 in Kibana logs when I try to create an Index pattern but I can see the HTTP 500 in httpd_logs :

ip_address - - [17/Oct/2019:12:09:19 +0200] *"POST /kibana/beta/elasticsearch/*%3A*/_search?ignore_unavailable=true HTTP/1.1" 500* 7328 "https://our_fqdn/kibana/beta/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

ip_address - - [17/Oct/2019:12:09:19 +0200] "POST /kibana/beta/elasticsearch/*/_search?ignore_unavailable=true HTTP/1.1" 500 7328 "https://our_fqdn/kibana/beta/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

So is the request from Kibana App wrong ? Is it a rewriting problem of the url ?

[lukas (Lukas Olson)]

What happens when you manually do a GET request on http://localhost:5601/kibana/beta/elasticsearch/*/_search ?

[ab2obek (Abdel)]

[root@kibanamachine ~]# curl -v -u kibana:kibana "http://localhost:5601/kibana/beta/elasticsearch/*%3A*/_search?ignore_unavailable=true"

• About to connect() to localhost port 5601 (#0)
• Trying localhost..
• Connected to 127.0.0.1 port 5601 (#0)
• Server auth using Basic with user 'kibana'

GET /kibana/beta/elasticsearch/%3A/_search?ignore_unavailable=true HTTP/1.1
Authorization: Basic a2liYW5hOmtpYmFuYQ==
User-Agent: curl/7.29.0
Host: 127.0.0.1
Accept: /

< HTTP/1.1 302 Found
< location: /kibana/beta/login?next=/kibana/beta/kibana/beta/elasticsearch/:/_search
< kbn-name: kibana

I forgot to mention that I have a readonlyrest plug-in installed for the basic auth but even when I deploy everything without including the RoR plugin, the 500 Error is here and when I try from the kibana machine i get a 302.