Connecting a local Kibana instance to a remote Elasticsearch cluster

[Trk_9987]

Hey folks.

We're investigating adding Kibana to one of our production Elasticsearch clusters. For security reasons, we are hesitant to run Kibana itself in production and are considering running kibana locally and pointing it to the ES cluster over HTTP.

I created a test ES cluster and configured elasticsearch.url to point to it's public IP. The only impact this made on the cluster was the creation of a .kibana index which was later populated with the index pattern I defined in Kibana.

We'd like to understand if there's any operational risks involved in implementing a setup like this. Is there a possibility that connecting kibana over HTTP like this could have an impact on cluster health? Is there any steps we should take to mitigate this like disabling transport communication from Kibana? We primarily want it for dev tools and some visualization.

Some other details which may be important to note:

We're actually running a custom proxy in front of the ES cluster which authenticates and routes to specific clusters based on custom HTTP headers. I had planned to use the Kibana config option for elasticsearch.customHeaders to facilitate this.

Thanks in advance for any advice!

[jbudz (Jon Budzenski)]

Both cases work operationally. I'll try and brain dump below.

In both cases there may be raw elasticsearch data going over the internet. If the Kibana server is in production, between the browser and Kibana server requests can be proxied (in cases like dev tools, discover, etc). If the server is local, it'll be between the kibana server and elasticsearch. You'll 100% want to enable TLS between both the Kibana server and browser, and Kibana server and elasticsearch.

Beyond that it's up to your security needs. If it's a no trust by default environment maybe you'll want to firewall the whole machine and throw it on a VPN.

To limit requests entirely when not in use you'll have to shut the server down. You can increase the health check delay (see below), but I wouldn't rely on it to stop all communication. I'm sure the server will keep sockets open if it can.

Cluster health shouldn't be impacted significantly due to it being local unless there's lots of Kibana installations instead of one. Kibana's server does a number of health checks on an interval. This can be limited with elasticsearch.healthCheck.delay. Otherwise the shift is mostly network travel time, Kibana will load quicker at the cost of elasticsearch queries taking some milliseconds longer.