Cluster Redundancy

[George_Smith]

Hi all,

I have a question regarding network redundancy with an Elasticsearch Cluster.
I am attempting to add redundancy to my cluster so that if a network adapter a node is using fails, it can use another network adapter to continue communicating with the cluster.

The way I am attempting to do this is by using the special value 0.0.0.0 for network.host to bind all nodes in the cluster to all available network adapters on the machine the node is running on, then explicitly specifying the IP addresses for network.bind_host and network.publish_host.

For example, say I have a 3 node cluster with each node running on a different machine.
The addresses/adapters available to the machines are both 10.0.2.xx and 10.0.3.xx .

Therefore for the machine where xx is 30, it's node YAML looks as follows:

network.host: 0.0.0.0
network.bind_host:
  - 10.0.2.30
  - 10.0.3.30
network.publish_host:
  - 10.0.2.30
  - 10.0.3.30

I would then expect that if I disabled the adapter on the machine with the address 10.0.2.30, the node should still be able to communicate with the cluster using the 10.0.3.30.
Unfortunately this is not the case, and when I disable the adapter the node is unable to communicate with the rest of the cluster.

I have noticed that using /_nodes/?pretty to check if the node is configured correctly, this information looks correct:

"network" : {
          "host" : "0.0.0.0",
          "bind_host" : [
            "10.0.2.30",
            "10.0.3.30"
          ],
          "publish_host" : [
            "10.0.2.30",
            "10.0.3.30"
          ]
        }

However the publish address only shows one value:

"transport" : {
        "bound_address" : [
          "10.0.2.30:9202",
          "10.0.3.30:9202"
        ],
        "publish_address" : "10.0.3.30:9202",
        "profiles" : { }
      },
      "http" : {
        "bound_address" : [
          "10.0.2.30:9201",
          "10.0.3.30:9201"
        ],
        "publish_address" : "10.0.3.30:9201",
        "max_content_length_in_bytes" : 104857600
      },

Could this be why my node fails to communicate with the cluster when 1 of the two network adapters available fails or is there something else I am missing?

Thanks in advance

[DavidTurner]

Hi there @George_Smith. Thanks for the questions. I think these docs will help to answer them:

You can specify a list of addresses for network.host and network.publish_host. [...] If you do this then Elasticsearch chooses one of the addresses for its publish address. This choice uses heuristics based on IPv4/IPv6 stack preference and reachability and may change when the node restarts. Ensure each node is accessible at all possible publish addresses.

Elasticsearch is based on redundancy at the node level. If a NIC (or disk, or PSU, or anything else within a node) fails then ES treats that as a whole-node failure and acts accordingly.

You can pull some system-level tricks to handle this kind of failure more gracefully (e.g. using RAID for disks or bonding for NICs) but that sort of thing is transparent to ES.

[George_Smith]

Hi @DavidTurner, many thanks for your timely response!

I had a feeling this was how Elasticsearch treated failures, looks like we'll need to investigate NIC bonding in this case.

Alternatively if NIC bonding is not appropriate for our usecase, I think we can use a workaround whereby we create n elasticsearch instances/nodes where n is the number of adapters on the machine, with each instance/node bound to 1 of the n adapters which should give us the desired redundancy characteristics, albeit at the cost of increased resource usage.

Thanks!

[DavidTurner]

I'm curious whether there's anything special about your environment that makes it worth putting so much effort into mitigating the risk of NIC failure. NICs do fail occasionally for sure, but do they fail unusually often for you?

[Christian_Dahlqvist]

@George_Smith

Alternatively if NIC bonding is not appropriate for our usecase, I think we can use a workaround whereby we create n elasticsearch instances/nodes where n is the number of adapters on the machine, with each instance/node bound to 1 of the n adapters which should give us the desired redundancy characteristics, albeit at the cost of increased resource usage.

This would be 3 separate nodes, each with their own data, and you would still lose one of them on NIC failure. I do not see how this is any better.