On ELK cluster new Kibana instance got error of unable to create Kibana index

[Mike_Z (Yang)]

It's an old cluster system built by someone else, and the existing Kibana instances are working fine.

I configured a new instance, and when trying to access it by browser, it returns an error saying unable to create Kibana index. I tried command "curl localhost:9200/_cat/indices" on the Kibana's machine, and indeed, the index does not exist. See more in section "Details".

Not sure if related to the root cause. I tried the same configuration of new Kibana on a virtual machine, for which all components are deployed on the same machine. And it works OK.

I will be very grateful to hints on how to resolve this issue, and have the below quetions in addition:

1. All the existing instances have kibana.yml files contain elasticsearch_url: "http://localhost:9200". Elasticsearch is deployed on other machines, so I don't understand why "localhost".

2. I wanted to enable logging to get more information on the error, but there was no entry of "logging.dest" in kibana.yml. I'm not sure whether this version doesn't have it, or I can manually add the entry anyway.

Please let me know if need more information.

Details:

1. Versions, and system architectures:

• Kibana = 4.1.2, deployed on two machines with GSLB (global service load balance);
• ElasticSearch = 1.4.4, deployed as a cluster consists of 3 machines;

2. Full error message displayed on browser:
   Fatal Error
   Kibana: Unable to create Kibana index ".kibana-newapp"
   Error: Unable to create Kibana index ".kibana-newapp"
   Error: MasterNotDiscoveredException[waited for [30s]]
   at respond (https://elk.cluster.domain/newapp/index.js?_b=7562:85289:15)
   at checkRespForFailure (https://elk.cluster.domain/newapp/index.js?_b=7562:85257:7)
   at https://elk.cluster.domain/newapp/index.js?_b=7562:83895:7
   at wrappedErrback (https://elk.cluster.domain/newapp/index.js?_b=7562:20902:78)
   at wrappedErrback (https://elk.cluster.domain/newapp/index.js?_b=7562:20902:78)
   at wrappedErrback (https://elk.cluster.domain/newapp/index.js?_b=7562:20902:78)
   at https://elk.cluster.domain/newapp/index.js?_b=7562:21035:76
   at Scope.$eval (https://elk.cluster.domain/newapp/index.js?_b=7562:22022:28)
   at Scope.$digest (https://elk.cluster.domain/newapp/index.js?_b=7562:21834:31)
   at Scope.$apply (https://elk.cluster.domain/newapp/index.js?_b=7562:22126:24)

[Marius_Dragomir]

The Kibana instance needs to point to the master node of that cluster. Check your elasticsearch_url and point it to that instance.

[Mike_Z (Yang)]

Thank you for your prompt help, @Marius_Dragomir, your hint works.

It was indeed the attribute 'elasticsearch_url' pointing to tribe's agent on localhost but not a master node of the cluster.

I found some backup files of kibana.yml pointing a Elasticsearch host, so I guess previously we created new Kibana instances with that node, and switched to localhost after done.

Please point out if I misunderstood anything here, and thank you again for your timely help.