Load multiple tables data from oltp to elasticsearch using logstash

What is the efficient way to load data from multiple tables (e.g. 10 to 15 tables from RDBMS) in to Elasticsearch . Note the RDBMS has records in millions so idea is to flatten the resultset and load in Elasticsearch .

Having statement in input configuration of logstash does not an efficient option as joining 10 to 15 tables in one go would not be a great idea . Alos some entities share One to Many relations ships.

E.g. Joining songs INNER JOIN song_folder for more tables would cause further performance bottlenck on source database.

jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "mysql"
parameters => { "favorite_artist" => "Beethoven" }
schedule => "* * * * *"
statement => "SELECT * FROM songs INNER JOIN song_folder using song_number;"
}
}
type or paste code here

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.