Sync Data from ES to MySQL


(Filipe Cunha) #1

i'm using elasticsearch to record a lot of messages. I want to create a ETL process with this ES data and create a DW in MySQL. How can i get data from ES, create a ETL process and insert it into MySQL DB? Any tool for that or is better use ,for example NodeJS or Python?


(Mark Walkom) #2

Logstash can do it with the JDBC input.


(David Pilato) #3

I think he wants to do that the other way.

Read from ES, write in MySQL.

We don't have yet a change API but for a one shot operation you can run a scroll query on elasticsearch to get all the docs. Then parse the JSON and write the SQL INSERT queries you need.


(Mark Walkom) #4

Ahh yeah, good point :stuck_out_tongue:


(Filipe Cunha) #5

Thank u. it is exactly what i want. But is not a scroll query too heavy? I want collect all data from one day (600k documents) and do a insert in MySQL. Do you think is better take small pieces and treat each one?


(David Pilato) #6

Do you think is better take small pieces and treat each one?

That's what scroll API is built for.
You scroll over x documents let say 200 by 200 docs.


(Filipe Cunha) #7

ok, i understand.
Thank you :slight_smile:


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.