Kibana High Availability

Elastic Stack Kibana
1

Dear All,
Been looking for achieving HA for Kibana service alone however did not come across any. In a traditional way assigning a VIP for multiple instance behind it will do the job but am not sure whether this will work for Kibana or not.
Did anyone tried configuring HA for Kibana? If yes then can someone through some light on how to achieve?

went through below links but they talk about load balancing & HA with ES

https://www.elastic.co/guide/en/kibana/current/production.html
https://www.elastic.co/blog/small-medium-or-large-scaling-elasticsearch-and-evolving-the-elastic-stack-to-fit
https://www.elastic.co/guide/en/kibana/current/production.html#load-balancing

Any thoughts?

1 Like
2

Say you have two Kibana instances A and B behind a load balancer. Are you looking at making a set up where

A] Either A or B is in use at any given time but not both. I.E. people are always connected to A, but if A goes down then everyone gets connected to B instead, until B goes down and then everyone gets connected to A instead.

Or

B] People are connected to A or B based on load/connections/whatever. A and B are both in use simultaneously but if A goes down everyone who was using A gets connected to B instead until A comes back up again.

I ask because I am currently looking at setting up scenario B. In fact I have it set up using HAProxy and it seems to work OK in very limited testing. But I was planning on posting here asking for advice on it because I have some concerns about it and can't find any documentation on doing it nor anyone having written up real world experience of it.

3

I am doing scenario B successfully, I manage to limit the load on each server and when I need to update/modify/restart/whatever I just remove one node from the load balancer and I don't notice when it happens as a user

4

when I need to update/modify/restart/whatever I just remove one node from the load balancer and I don't notice when it happens as a user

Is there a specific reason you remove the node from HAProxy configuration rather than using on a health check to make HAProxy not route to a Kibana instance which isn't available? At what point do you put it back in the HAProxy configuration? (When it's been upgraded? When all Kibana nodes have been upgraded?)

Am I correct in assuming that all your Kibana nodes are storing their configuration in the same index? The concerns I have are around multiple Kibana nodes in use at the same time all storing their configuration in the same index. If the Kibana nodes aren't all using the same index not then the index patterns and visualisations and such are going to be different depending on which Kibana node they get routed to and that's no use.

5

Is there a specific reason you remove the node [...] ?

I also have the healthchecks, but I have made mistakes in the past where when I get Kibana back up and there is an ES node in the cluster that didn't upgrade and Kibana while running, is not working.
Where I come from we have a saying that translates to "Wearing a belt and suspenders", you can never be too careful

Am I correct in assuming that all your Kibana nodes are storing their configuration in the same index?

Yes, there is just one kibana index that both nodes use, hasn't been a trouble for me, not sure how it would be a problem, the likeliness of two users on 2 instances trying to do 2 contradicting things on the same microsecond sounds unlikely and even in a split-brain situation or something like that the Kibana instances only talk to a local ES node, not master eligible with no data. so if that server were to get disconnected from the rest of the cluster, it wouldn't be able to do anything

Ps. Not using HAProxy, using hardware load balancers but the same logic applies here

6

It's good to hear that you're doing multiple Kibana in use at the same time all using the same index without problems. With regard upgrading Kibana, is there anything to watch out for there? Like one node is upgraded from Kibana N.0 to N.1 and then put back in to service whilst the other nodes are still on N.0, are all the Kibana isntances happy with that?

7

@mikewillis

Yeah, if you have kibana running and then go around, upgrading elastic nodes without restarting kibana, kibana doesn't complain.

This is at least true when running Kibana on a server with an elastic node (no data, no master eligible) that is its only contact to the cluster
...... no idea what happens otherwise

However if you upgrade some and then restart Kibana it is a whole other story.

so what I ususally do is keep this on hand Upgrade to Elastic 8.11.1 | Elastic Installation and Upgrade Guide [8.11] | Elastic , since this contains the upgrade order

What I will then do is :

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "none"
}
}

so that when the node leaves the cluster while upgrading, the other nodes don't try to allocate the indices to themselves

Then I upgrade a single node
when I have upgraded the node and confirmed it has rejoined the cluster I will run

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}

so that the cluster "heals" itself again, with the node present, wait until I am happy with the index allocation on the cluster and then take the next node

Takes quite a lot of time with a large cluster (I have 15 nodes in my main cluster and I am considering scripting this via ansible)

when doing this order I recomend just doing the Kibana servers last

8

@gborg

Your answer makes me realise I asked my question about Kibana upgrades in a way which didn't actually ask what I meant to. What I'm wondering is whether an upgraded Kibana node might make modifications to the Kibana index which are a problem for Kibana instances that haven't been upgraded. E.g. Node A gets updated to Kibana N.1 and put back in service, Node A does something to the index, and then Nodes B and C which are still running Kibana N.0 have problems because of modifications made to the index by Node A. Have you encountered anything like that? Is there any difference in how you handle major and minor version upgrades to Kibana behind a load balancer?

On the cluster I work with we upgrade Elasticsearch on all the nodes before upgrading Kibana and like you we run Kibana configured to talk to an Elasticsearch node on the same server that doesn't hold data and can't be master. We have 37 Elasticsearch nodes and have created an Ansible playbook to handle upgrades. Ansible doesn't really make the upgrading quicker but does make it much easier!

9

@mikewillis

Ansible doesn't really make the upgrading quicker but does make it much easier!

Yeah, that is what I am hoping for :slight_smile:
I would like to be able to push 1 button and then just do something else what that is being crunched in the background

[....] ich are still running Kibana N.0 have problems because of modifications made to the index by Node A. Have you encountered anything like that?

I haven't and I don't think it should happen, these upgrades are supposed to be smooth

Is there any difference in how you handle major and minor version upgrades to Kibana behind a load balancer ?

I actually haven't upgraded between major versions on my current setup but in my memory it wasn't difficult to do so on the elastic side at least so I would guess you could do it the same way

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.