Cross Account (AWS) cluster

[chaib (Giovanni Toledo)]

Is it possible to have an ES cluster with nodes in two different accounts (AWS)?

Scenario:

• Account A with VPC peering to Account B.
• Account B nodes find master in Account A but return a message that failed to send a request asking to join the cluster.
• Account A range of IP's: 10.0.0.0/16
• Account B range of IP's: 172.72.0.0/16

Goal:

The purpose of this architecture is to transfer all data in the current cluster A to B. Snapshot is not a valid strategy because of the fact that the cluster A is still consuming data. Is there a better way to do this?

[Magnus_Kessler]

There is no inherent reason why an Elasticsearch cluster cannot span two network segments (172.72.0.0/16 and 10.0.0.0/16 in your example). However, you need to ensure that the cluster internal traffic can pass unhindered in both directions between the network segments. Please check your network settings again.

Having said that, if your ultimate goal is to transfer the data from cluster A to cluster B, you could initialise cluster B with a snapshot from cluster A, giving you a cluster that is close to the still evolving state. Afterwards, you can transfer additional changes from those indices that still take on new data from cluster A with the Reindex API. The Reindex API allows the use of queries, and you can transfer only the subset of data that has been changed after a certain point in time.

When cluster B has most of the data, switch traffic over to this new cluster, and then perform a last reindex run to mop up any documents that have not net been indexed on cluster B. During this period, a very small amount of documents may still be out of date on cluster B, but should be back-filled quickly.