Syncing Huge DataSet From MySQL to Elasticsearch

[Aswini_Kumar_Rout]

Hi Team,

We have one requirement to use Elasticsearch in our legacy application which has MySQL datbase and the size of the DB is around 300 GB and with 200 or more tables. So, here I am bit confused that - which would be the BEST way to do the sync, I found same kind of questions which are posted earllier but I am not fully clear on the approaches which are explained -

Let me explain my requirement properly -

1. Sync all the Data from MySQL to Elasticsearch on Daily basis - For this I am confused with below approaches -

   • Using Logstash with a cron

   • Using Java application with a cron

   For the seprate Java application below are the refrences -

Medium Link - How to synchronize MySQL Database with ElasticSearch And perform data querying in a Spring Boot Application | by YASSINE SABRI | Medium

GIT Link - GitHub - yassinesabri/spring-mysql-elastic: Dockerized Spring POC implementing the synchronization and the data querying between MySql and ElasticSearch engine

Kindly suggest the BEST way considering the size of data.

2. Java APIs to get data from Elasticsearch with complex Stored Procedures which are currently running in our application to show some dashboard and data analysis (We are thinking of Kibana for this).

Above is the requirement, Kindly suggest the best way to implement this considering the Performance and data.

Thanks