Consolidating/aggregating ES documents on one ELK stack

Elastic Stack Elasticsearch
#1

Hi all,

I'm trying to solve the following problem: I have data in RDSs (Aurora MySQL), a subset of which I'm going to map to Elasticsearch using the AWS Database Migration Service. Unfortunately, we have full deployments in 4 AWS regions, but need to consolidate the output onto one Elasticsearch instance. The source databases aren't publicly accessible and AWS DMS doesn't support Elasticsearch targets in different regions, so I need to forward all the ES documents to a primary ES instance in one region. Here's a simplified diagram:

region 1
region 11536×1248 102 KB

The forwarding ES doesn't need to keep anything; all searching will be done on the primary ES. What's the best way to consolidate documents from multiple Elasticsearch instances onto this single primary instance?

Thanks!

#2

I'd look at either CCR or CCS to do what you want, would make things easier.

#3

Thanks for the suggestion. Unfortunately, AWS's elasticsearch service doesn't support CCR or CCS across AWS regions, and the AWS Database Migration Service only supports the ES service, not a self-managed stack. Perhaps using CCR is still the way to go, but this would require mapping the RDS data to ES by some other method.

#4

You can always use Elastic Cloud, which can run on AWS and is available in their Marketplace, which will provide you with a heap more functionality.

#5

As far as I can tell, Elastic Cloud isn't a supported DMS target, unfortunately.

#6

Not sure what that means sorry.

#7

DMS is the AWS Database Migration Service, which provides a fairly simple way to replicate a subset of the data in the relational DBs to Elasticsearch.

#8

Ah right, what about using Logstash with the JDBC input then?

#9

That might be the way to go, thanks.

#10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.