Trying to implement CDC transfer from MongoDB to Elasticsearch 8.4

Elastic Search
1

We are trying to implement CDC transfer from MongoDB to Elasticsearch 8.4. Do we have to manually create index mappings for each Mongo collection or can we enable dynamic mappings on create within Elasticsearch

2

Hi @Venkatesh_Guruprasad,

First, can you clarify what "CDC" stands for? I'm unsure if that adds important context for your question.

Second, are you using Elasticsearch API, or Elastic Enterprise Search? If you're using Elasticsearch directly, we may want to re-tag this post to stack-elasticsearch.

However, you may be interested to know that Enterprise Search has been developing a connector specifically for syncing MongoDB data to Elasticsearch. See: Elastic connectors | Elastic Enterprise Search documentation [8.4] | Elastic
This feature is in "technical preview", which means we're actively developing on and iterating on this. Stay tuned as future releases come out.

If you'd like to be an early adopter and provide us some feedback, we'd be very interested to hear it.

1 Like
3

@Venkatesh_Guruprasad we'd love to get your use case details to see how our MongoDB connector can facilitate your work flow!

4

We are trying to move modified data in MongoDB cluster to an Elasticsearch instance on the cloud. CDC we referring is for Change Data Capture. If you have a beta version of the connector being built would like to gain access to that. We are in the process of trying to find a solution.

5

Just to clarify @Venkatesh_Guruprasad - this is about MondoDB change streams, right? They're watching change events on the collection and these are what you want then be exported to another data source - to Elastic?

6

Yes that's correct. We want our sync with Mongo in Elastic as close as possible.

7

We are using Mongo as our operational DB and Elastic as the analytical DB. We want to do transformation for the analytical pipeline using ELT rather than ETL. By doing ELT we are localizing the transforms within Elastic & removing any other tool in between. For ELT to happen we want to the change streams from Mongo sync with Elastic always.

8

in 8.4 we released a technical preview of MongoDB connector client 8.4.0 release notes | Elastic Enterprise Search documentation [8.4] | Elastic
we'd love your feedback on this! The code for this connector client to Mongo can be found in our open code repo GitHub - elastic/connectors-ruby: Official Connector Clients for Elastic Elasticsearch, Enterprise Search, App Search and Workplace Search, you just need to deploy it on self managed infrastructure and connect to Enterprise Search. We also have a native connector hosted in Enterprise Search on the short term roadmap. If you'd like to try that out, i can DM you to set up a user feedback session

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.