How to consume a Elastic indexs between different Clouds

Elastic Stack Elasticsearch
1

We run Elasticsearch using a self-hosting solution from GCP. Our services consume a BD in Elastic to work correctly. A new client asks us if it is possible to consume their Elastic BD directly from their Elastic self-hosting solution from AWS. So there's no need to duplicate or re-index the base. Can someone help with this?

2

Can you provide more context? It is not clear what is the issue or what you want to do.

You have a service that uses Elasticsearch? Where is the data that your client wants?

3

Sure. We have a product that uses Elasticsearch and ES DB. A normal use case would be our final client uploading their data via our UI which is then stored at ES DB at our self-hosting solution from GCP. But in this case, this new client already has all their data on ES DB but at AWS cloud. So, to summarize, our ES application on GCP cloud needs to access their ES DB at AWS cloud.

4

What do you mean by ES DB? It is an Index? It is Elasticsearch App Search? It is not clear.

If you have a product that uses Elasticsearch behind the scenes, you would need to change your code to make it able to connect to another Elasticsearch Instance, I'm not sure what is the issue here.

Or do you want to make your GCP elasticsearch cluster access your client Elasticsearch cluster on AWS?

5

By DB I mean Index. We need to access their index that is hosted on AWS. So, I would rephrase it as "access an Elasticsearch index that is hosted on AWS from an Elasticsearch cluster that is hosted on GCP"

6

I've found this guideline, but I don't know if it makes sense:

Here is a summary of the steps to access an Elasticsearch index on AWS from an Elasticsearch cluster on GCP, ensuring compatibility between the tokenization settings:

  1. Set up a VPN or direct connect between your GCP VPC and the VPC that hosts the Elasticsearch index on AWS. This will allow your GCP cluster to communicate with the Elasticsearch index on AWS.
  2. Add the appropriate network rules to the GCP firewall and/or the security group on the Elasticsearch cluster to allow access to the Elasticsearch index on AWS.
  3. Review the tokenization settings on the Elasticsearch index on AWS and the GCP application, and make sure that they are compatible. Consult the documentation for both Elasticsearch and the GCP application, and experiment with different settings to find the best configuration.
  4. Use a common tokenization scheme on both the Elasticsearch index on AWS and the GCP application, if possible, to ensure compatibility.
  5. Use Elasticsearch Mappings API to verify and update the tokenization settings on the Elasticsearch index, if needed.
  6. Test the connectivity and access to the Elasticsearch index on AWS from your GCP cluster. Use the Elasticsearch APIs or a tool such as Kibana to connect to the Elasticsearch index on AWS and verify that the data can be accessed and processed correctly.
  7. If the tests are successful, you can start using the Elasticsearch index on AWS from your GCP cluster in your production environment.

Note: These steps are general guidelines and may vary depending on your specific setup and requirements. It is recommended to consult with an expert or refer to the Elasticsearch and AWS/GCP documentation for more detailed instructions.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.