What is the efficient way to load data from multiple tables (e.g. 10 to 15 tables from RDBMS) in to Elasticsearch . Note the RDBMS has records in millions so idea is to flatten the resultset and load in Elasticsearch .
Having statement in input configuration of logstash does not an efficient option as joining 10 to 15 tables in one go would not be a great idea . Alos some entities share One to Many relations ships.
E.g. Joining songs INNER JOIN song_folder for more tables would cause further performance bottlenck on source database.
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "mysql"
parameters => { "favorite_artist" => "Beethoven" }
schedule => "* * * * *"
statement => "SELECT * FROM songs INNER JOIN song_folder using song_number;"
}
}
type or paste code here