I'm migrating MYSQL data to ElasticSearch using logstash.
The table has more than 9 crore records. I have tested the query in mysql workbeanch which is taking 0.032 sec to execute.
When i run it from logstash, its taking more than 600 sec.
What will be the reason ?
Could you guys please help me ?
This is my logtash conf file.
jdbc_driver_library => "mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/database"
jdbc_user => "abc"
jdbc_password => "xyz"
statement_filepath => "select * from table_name"
I have a pagination in the query using primary key. for single fetch it will take 100000 records.
page = 100000
select * from table_name where id between 0 and page;
this page will increment by 100000 using shell script.