i'm using elasticsearch to record a lot of messages. I want to create a ETL process with this ES data and create a DW in MySQL. How can i get data from ES, create a ETL process and insert it into MySQL DB? Any tool for that or is better use ,for example NodeJS or Python?
Logstash can do it with the JDBC input.
I think he wants to do that the other way.
Read from ES, write in MySQL.
We don't have yet a change API but for a one shot operation you can run a scroll query on elasticsearch to get all the docs. Then parse the JSON and write the SQL INSERT queries you need.
Ahh yeah, good point 
Thank u. it is exactly what i want. But is not a scroll query too heavy? I want collect all data from one day (600k documents) and do a insert in MySQL. Do you think is better take small pieces and treat each one?
Do you think is better take small pieces and treat each one?
That's what scroll API is built for.
You scroll over x documents let say 200 by 200 docs.
ok, i understand.
Thank you 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.