Kibana "Fails to load indices" when trying to set up an index pattern in Management tab

[ljh33txm]

Hey all, I recently upgraded my elastic stack to 6.6.2 from 5.6.x. I got everything running but when I load into Kibana's Management tab to create my index patterns it says "Failed to load indices". The strange thing is when I go to Management --> Elasticsearch --> Index Management, all of my indices are properly listed. Why would one tab of Kibana be able to see it fine but the other not? Thanks!

Setup info: Running on Windows 10 server. Elasticsearch installed with MSI as Local System Service, Kibana set up with NSSM as Local System Service. Created URL Rewrite rules for both in the IIS setup. Am able to properly communicate with both via the configured website.

[bhavyarm]

@mattkime can you please grab this one?

Thanks,
Bhavya

[ljh33txm]

@bhavyarm Did anyone get a chance to look at this? The upgrade is a bit time sensitive. Thanks!!

[Christian_Dahlqvist]

Where is IIS in the flow? Does it work if you exclude IIS?

[ljh33txm]

@Christian_Dahlqvist After Installing Elasticsearch via MSI and Kibana via zip, I wrapped the folder location in a website with IIS. I then set up 2 URL rewrite rules:

The Kibana rule (applied to the Kibana install folder) is Pattern: (.), Rewrite URL: http://{server IP}:5601/{R:1}
The Elasticsearch rule (applied to the Elasticsearch install folder) is Pattern: (.), Rewrite URL: http://127.0.0.1:9200/{R:1}

This seems to work as I can access Kibana and write logs to Elasticsearch via the DNS entry our company created. In fact, as I stated in the first post, under Management --> Elasticsearch --> Index Management in Kibana it correctly shows all the log indices that have been written to Elasticsearch so that all seems to be working. It is only in the Management --> Kibana --> Create Index Pattern screen where I immediately get 2 pop up errors and see no indices:

"Failed to load remote clusters"
"Failed to load indices"

[Christian_Dahlqvist]

Does Kibana connect to Elasticsearch via IIS? If so, does the behaviour change if you connect Kibana directly to Elasticsearch?

[ljh33txm]

@Christian_Dahlqvist No, Im only using IIS as a wrapper so the outside word can communicate with it (URL Rewrite rules), pretty sure both of them are running in localhost as far as they are concerned.

[Christian_Dahlqvist]

@ljh33txm

pretty sure both of them are running in localhost

Can you show us the kibana.yml file just to be sure?

[ljh33txm]

@Christian_Dahlqvist I have most everything commented out except:

server.port: 5601
server.basePath: "/custompath" #This is reflected in the IIS setup
server.rewriteBasePath: false
server.name: "TheServerName"

[Christian_Dahlqvist]

Can you go to the Console under Dev Tools in Kibana and run GET /.kibana*/_settings?

[ljh33txm]

@Christian_Dahlqvist Sure thing, here it is:

{
".kibana_1" : {
"settings" : {
"index" : {
"number_of_shards" : "1",
"auto_expand_replicas" : "0-1",
"provided_name" : ".kibana_1",
"creation_date" : "1553614167821",
"number_of_replicas" : "0",
"uuid" : "NSChLFobQ3CpDQNCEGxzXQ",
"version" : {
"created" : "6060299"
}
}
}
}
}

[ljh33txm]

@Christian_Dahlqvist, @bhavyarm Any updates on this? Need me to provide more info? Our logs have been down now for a week.

[mattkime (Matthew Kime)]

ljh33txm -

If you pop open your browser's inspector, do you see a failed http request?
Similarly, do you see an error in the Kibana logs?

[ljh33txm]

@mattkime Not a web developer so bear with me, I think so?

[mattkime (Matthew Kime)]

@ljh33txm Yes, those are failed http requests. Do you see matching failures in the kibana logs? I wonder if the urls are being rewritten incorrectly and I'm curious if they're making it to kibana.

[m2web]

We were seeing the same thing with Kibana 6.7.0 on Windows 2016 Server. Looking at the Event Logs in Windows I noted the following:
HttpException
A potentially dangerous Request.Path value was detected from the client (). at System.Web.HttpRequest.ValidateInputIfRequiredByConfig() at System.Web.HttpApplication.PipelineStepManager.ValidateHelper(HttpContext context) http://OurURL/elasticsearch//_search?ignore_unavailable=true/elasticsearch/*/_search

The fix was to add the following to the IIS web.config:
<system.web>
<httpRuntime requestPathInvalidCharacters="" / >
</system.web>

Stop and restart the web in IIS and we were good.

[ljh33txm]

@m2web In the Elasticsearch or Kibana web.config? Thanks!

[ljh33txm]

@m2web Actually, scratch that, got it to work in the Kibana web.config. Thank you so much!!