Consolidating/aggregating ES documents on one ELK stack

Hi all,

I'm trying to solve the following problem: I have data in RDSs (Aurora MySQL), a subset of which I'm going to map to Elasticsearch using the AWS Database Migration Service. Unfortunately, we have full deployments in 4 AWS regions, but need to consolidate the output onto one Elasticsearch instance. The source databases aren't publicly accessible and AWS DMS doesn't support Elasticsearch targets in different regions, so I need to forward all the ES documents to a primary ES instance in one region. Here's a simplified diagram:

The forwarding ES doesn't need to keep anything; all searching will be done on the primary ES. What's the best way to consolidate documents from multiple Elasticsearch instances onto this single primary instance?


I'd look at either CCR or CCS to do what you want, would make things easier.

Thanks for the suggestion. Unfortunately, AWS's elasticsearch service doesn't support CCR or CCS across AWS regions, and the AWS Database Migration Service only supports the ES service, not a self-managed stack. Perhaps using CCR is still the way to go, but this would require mapping the RDS data to ES by some other method.

You can always use Elastic Cloud, which can run on AWS and is available in their Marketplace, which will provide you with a heap more functionality.

As far as I can tell, Elastic Cloud isn't a supported DMS target, unfortunately.

Not sure what that means sorry.

DMS is the AWS Database Migration Service, which provides a fairly simple way to replicate a subset of the data in the relational DBs to Elasticsearch.

Ah right, what about using Logstash with the JDBC input then?

That might be the way to go, thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.