Kibana “Failed to load indices” and “Failed to load remote clusters”

Hi everyone,

I recently installed an ElasticSearch cluster (3 mdi machines), Kibana (1 machine), both on v7.3.0 and everything seems to be working.

My problem is that when I click on "Create Index Pattern" from Kibana's management tab, it notifys "Failed to load indices" and "Failed to load remote clusters" but i can see all indexes on the index pattern form, I can load them and create a pattern.

image

The failed to load error :

image

In the "network" tab of page inspection on chrome I can see two requests that goes into a "http 500 error"

The urls are :
https://our_fqdn/kibana/beta/elasticsearch/*/_search?ignore_unavailable=true
https://our_fqdn/kibana/beta/elasticsearch/*%3A*/_search?ignore_unavailable=true

Some infos about my setup

FQDN ** ----> ** httpd** ---->** Kibana / Elastic (port 5601)

The config files :

elasticsearch.yml

network.host: 127.0.0.1, eth2_ip
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ip_of_master1,ip_of_master2,ip_of_master3
indices.fielddata.cache.size: 50%
reindex.remote.whitelist: "master1:9200, master2:9200, master3:9200"

kibana.yml

server.port: "5601"
server.host: "IP_of_the_server"
server.basePath: "/kibana/beta"
server.rewriteBasePath: false
elasticsearch.hosts: "http://localhost:9200"
elasticsearch.username: "user"
elasticsearch.password: "pass"

httpd config

<Location /kibana/beta>
ProxyPass http://kibana_ip:90
ProxyPassReverse http://kibana_ip:90
</Location>

I really can't find what is wrong so that i'm getting this error.
What i've tried :

  • I tried to activate the rewrite.Path (negative)
  • I tried the to comment reindex.remote.whitelist: "master1:9200, master2:9200, master3:9200" (negative)
  • I tried the curl locally and the only curl seems to work is
    curl -XPOST "http://localhost:9200/elasticsearch/*/_search?ignore_unavailable=true"
    So maybe the base request should be
    https://our_fqdn/elasticsearch/*/_search?ignore_unavailable=true without the "kibana/beta" part ?

If anyone got an idea it would be greatly appreciated.

PS: For confidentiality issues I replaced all the IPs with some names like for "master1", "master2" but in my config file it's actually IP addresses :slight_smile:

Hmm, are you seeing any errors in the Kibana server console?

You mentioned curl -XPOST "http://localhost:9200/elasticsearch/*/_search?ignore_unavailable=true" is successful, but do you really mean http://localhost:9200/*/_search?ignore_unavailable=true"(without the elasticsearch portion)? If the former is working for you then that is certainly confusing.

Hi Lukas,

Actually the curl that worked in the Kibana console is the one sent as a POST when i try to create an index pattern. Here is the result in Kibana console :

There is no HTTP 500 in Kibana logs when I try to create an Index pattern but I can see the HTTP 500 in httpd_logs :

ip_address - - [17/Oct/2019:12:09:19 +0200] *"POST /kibana/beta/elasticsearch/*%3A*/_search?ignore_unavailable=true HTTP/1.1" 500* 7328 "https://our_fqdn/kibana/beta/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

ip_address - - [17/Oct/2019:12:09:19 +0200] "POST /kibana/beta/elasticsearch/*/_search?ignore_unavailable=true HTTP/1.1" 500 7328 "https://our_fqdn/kibana/beta/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

So is the request from Kibana App wrong ? Is it a rewriting problem of the url ?

What happens when you manually do a GET request on http://localhost:5601/kibana/beta/elasticsearch/*/_search ?

[root@kibanamachine ~]# curl -v -u kibana:kibana "http://localhost:5601/kibana/beta/elasticsearch/%3A/_search?ignore_unavailable=true"

  • About to connect() to localhost port 5601 (#0)
  • Trying localhost..
  • Connected to 127.0.0.1 port 5601 (#0)
  • Server auth using Basic with user 'kibana'

GET /kibana/beta/elasticsearch/%3A/_search?ignore_unavailable=true HTTP/1.1
Authorization: Basic a2liYW5hOmtpYmFuYQ==
User-Agent: curl/7.29.0
Host: 127.0.0.1
Accept: /

< HTTP/1.1 302 Found
< location: /kibana/beta/login?next=/kibana/beta/kibana/beta/elasticsearch/:/_search
< kbn-name: kibana

I forgot to mention that I have a readonlyrest plug-in installed for the basic auth but even when I deploy everything without including the RoR plugin, the 500 Error is here and when I try from the kibana machine i get a 302.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.