# Migration of data from Elasticsearch to Databases (MongoDB, MySQL) **URL:** \ **Category:** Elasticsearch\ **Created:** [June 18, 2024, 5:57pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669 "2024-06-18T17:57:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [June 18, 2024, 5:57pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/1 "2024-06-18T17:57:20Z")
Brief Principle: The real-time incremental data is written to both the old and new storage engines (ES and DB). Then data from the old storage engine (ES) is migrated to the new storage engine (DB). After that finish, data consistency is verified. Finally, read data are switched to the new storage system(DB). This is my first time performing a data migration, so I would appreciate any reviews, suggestions, or references to mature solutions. Steps: 1. **Create Database Tables** : Based on the schema of ES, create tables in MongoDB and MySQL respectively. 2. **Dual Write Incremental Data to ES and DB** : Write the real-time incremental data to both ES and DB simultaneously. 3. **Migrate Historical Data from ES to DB** : Write all data from ES to DB using the Logstash tool. 4. **Data Verification** : Traverse all data in ES, ensuring that each data can be found in DB. If the data is inconsistent, modify the data in DB according to the data in ES. 5. **Switch Query Engine** : Switch the query engine from ES to DB. 6. **Monitor Data Read/Write** : Check if the read/write operations are normal, especially whether the read performance meets the requirements. 7. **Stop Dual Write and remove ES Cluster** : Stop writing incremental data to ES and remove the ES cluster. ---
**Author:** ![jessgarson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jessgarson/32/129841_2.png) [@jessgarson](https://discuss.elastic.co/u/jessgarson)\ **Post date:** [June 18, 2024, 8:05pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/2 "2024-06-18T20:05:53Z")
Thanks for reaching out, @Andy_Cong. Here are a few resources that may be helpful to take a look at: - [MongoDB native connector tutorial | Enterprise Search documentation [8.14] | Elastic](https://www.elastic.co/guide/en/enterprise-search/current/mongodb-start.html) - [Elastic Microsoft SQL connector reference | Enterprise Search documentation [8.14] | Elastic](https://www.elastic.co/guide/en/enterprise-search/current/connectors-ms-sql.html) - [Elastic MongoDB connector reference | Enterprise Search documentation [8.14] | Elastic](https://www.elastic.co/guide/en/enterprise-search/current/connectors-mongodb.html) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2024, 8:36pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/3 "2024-06-18T20:36:59Z")
> [@Andy\_Cong](#): > > **Dual Write Incremental Data to ES and DB** : Write the real-time incremental data to both ES and DB simultaneously. Relatively easy if you are only ingesting new immutable data. If you need to deal with deletes and updates it gets more complicated. > [@Andy\_Cong](#): > > **Migrate Historical Data from ES to DB** : Write all data from ES to DB using the Logstash tool. Logstash was designed to move data into Elasticsearch. I do not think it is a good chioce when replicating or moving data in the other direction. You probably need to develop scripts or an application to do this as the data likely need to be transformed as well. > [@Andy\_Cong](#): > > **Data Verification** : Traverse all data in ES, ensuring that each data can be found in DB. If the data is inconsistent, modify the data in DB according to the data in ES. If data is constantly being modified this can be tricky to do without downtime. ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [June 19, 2024, 5:26pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/4 "2024-06-19T17:26:33Z")
Very nice！ Thank you ！！ ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [June 20, 2024, 4:09pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/5 "2024-06-20T16:09:02Z")
> [@Christian\_Dahlqvist](#): > > Relatively easy if you are only ingesting new immutable data. If you need to deal with deletes and updates it gets more complicated. You are right, very complicated！The real-time incremental data need update。 ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [June 20, 2024, 4:11pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/6 "2024-06-20T16:11:19Z")
> [@Christian\_Dahlqvist](#): > > Logstash was designed to move data into Elasticsearch. I do not think it is a good chioce when replicating or moving data in the other direction. You probably need to develop scripts or an application to do this as the data likely need to be transformed as well. Good suggestion, I'm going to write a script to migrate history data. ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [June 20, 2024, 4:17pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/7 "2024-06-20T16:17:29Z")
> [@Christian\_Dahlqvist](#): > > If data is constantly being modified this can be tricky to do without downtime. Yes, very careful in doing this, so we need experienced people to review the solution and give some idea. ---
**Author:** ![Andy\_Cong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_cong/32/134851_2.png) [@Andy\_Cong](https://discuss.elastic.co/u/Andy_Cong)\ **Post date:** [June 20, 2024, 4:18pm UTC](https://discuss.elastic.co/t/migration-of-data-from-elasticsearch-to-databases-mongodb-mysql/361669/8 "2024-06-20T16:18:54Z")
Thanks for you suggestions!! I will consider your suggestions。