Consolidating/aggregating ES documents on one ELK stack

[Asher]

Hi all,

I'm trying to solve the following problem: I have data in RDSs (Aurora MySQL), a subset of which I'm going to map to Elasticsearch using the AWS Database Migration Service. Unfortunately, we have full deployments in 4 AWS regions, but need to consolidate the output onto one Elasticsearch instance. The source databases aren't publicly accessible and AWS DMS doesn't support Elasticsearch targets in different regions, so I need to forward all the ES documents to a primary ES instance in one region. Here's a simplified diagram:

The forwarding ES doesn't need to keep anything; all searching will be done on the primary ES. What's the best way to consolidate documents from multiple Elasticsearch instances onto this single primary instance?

Thanks!

[warkolm (Mark Walkom)]

I'd look at either CCR or CCS to do what you want, would make things easier.

[Asher]

Thanks for the suggestion. Unfortunately, AWS's elasticsearch service doesn't support CCR or CCS across AWS regions, and the AWS Database Migration Service only supports the ES service, not a self-managed stack. Perhaps using CCR is still the way to go, but this would require mapping the RDS data to ES by some other method.

[warkolm (Mark Walkom)]

You can always use Elastic Cloud, which can run on AWS and is available in their Marketplace, which will provide you with a heap more functionality.

[Asher]

As far as I can tell, Elastic Cloud isn't a supported DMS target, unfortunately.

[warkolm (Mark Walkom)]

Not sure what that means sorry.

[Asher]

DMS is the AWS Database Migration Service, which provides a fairly simple way to replicate a subset of the data in the relational DBs to Elasticsearch.

[warkolm (Mark Walkom)]

Ah right, what about using Logstash with the JDBC input then?

[Asher]

That might be the way to go, thanks.