I 'am using Elasticsearch 5.1.1 , logstash 5.1.1 ,I imported 3 millions rows from sqlserver into elastic via logstash in 2 hours
I have 1 single windows machine with 4GB Ram , core I 3 is there any additional configurations should I add to speed up the importing ?
Logstash Configurations
input {
jdbc {
jdbc_driver_library => "D:\Usefull_Jars\sqljdbc4-4.0.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://192.168.5.14:1433;databaseName=DataSource;integratedSecurity=false;user=;password=;"
jdbc_user => ""
jdbc_password => ""
statement => "SELECT * FROM RawData"
jdbc_fetch_size => 1000
}
}
output {
elasticsearch {
hosts => "localhost"
index => "testdata"
document_type => "testfeed"
document_id => "%{id}"
flush_size => 512
}
}