On ELK cluster new Kibana instance got error of unable to create Kibana index

Elastic Stack Kibana
1

It's an old cluster system built by someone else, and the existing Kibana instances are working fine.

I configured a new instance, and when trying to access it by browser, it returns an error saying unable to create Kibana index. I tried command "curl localhost:9200/_cat/indices" on the Kibana's machine, and indeed, the index does not exist. See more in section "Details".

Not sure if related to the root cause. I tried the same configuration of new Kibana on a virtual machine, for which all components are deployed on the same machine. And it works OK.

I will be very grateful to hints on how to resolve this issue, and have the below quetions in addition:

  1. All the existing instances have kibana.yml files contain elasticsearch_url: "http://localhost:9200". Elasticsearch is deployed on other machines, so I don't understand why "localhost".

  2. I wanted to enable logging to get more information on the error, but there was no entry of "logging.dest" in kibana.yml. I'm not sure whether this version doesn't have it, or I can manually add the entry anyway.

Please let me know if need more information.

Details:

  1. Versions, and system architectures:
  • Kibana = 4.1.2, deployed on two machines with GSLB (global service load balance);
  • ElasticSearch = 1.4.4, deployed as a cluster consists of 3 machines;
  1. Full error message displayed on browser:
    Fatal Error
    Kibana: Unable to create Kibana index ".kibana-newapp"
    Error: Unable to create Kibana index ".kibana-newapp"
    Error: MasterNotDiscoveredException[waited for [30s]]
    at respond (https://elk.cluster.domain/newapp/index.js?_b=7562:85289:15)
    at checkRespForFailure (https://elk.cluster.domain/newapp/index.js?_b=7562:85257:7)
    at https://elk.cluster.domain/newapp/index.js?_b=7562:83895:7
    at wrappedErrback (https://elk.cluster.domain/newapp/index.js?_b=7562:20902:78)
    at wrappedErrback (https://elk.cluster.domain/newapp/index.js?_b=7562:20902:78)
    at wrappedErrback (https://elk.cluster.domain/newapp/index.js?_b=7562:20902:78)
    at https://elk.cluster.domain/newapp/index.js?_b=7562:21035:76
    at Scope.$eval (https://elk.cluster.domain/newapp/index.js?_b=7562:22022:28)
    at Scope.$digest (https://elk.cluster.domain/newapp/index.js?_b=7562:21834:31)
    at Scope.$apply (https://elk.cluster.domain/newapp/index.js?_b=7562:22126:24)
2

The Kibana instance needs to point to the master node of that cluster. Check your elasticsearch_url and point it to that instance.

3

Thank you for your prompt help, @Marius_Dragomir, your hint works.

It was indeed the attribute 'elasticsearch_url' pointing to tribe's agent on localhost but not a master node of the cluster.

I found some backup files of kibana.yml pointing a Elasticsearch host, so I guess previously we created new Kibana instances with that node, and switched to localhost after done.

Please point out if I misunderstood anything here, and thank you again for your timely help.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.