I am using logstash 6.1.0 to make a data pipeline to stash data from DEV server(where mysql data is present using jdbc input plugin) to TEST server( where elastic search in installed).
The config file for logstash is as follows:
input {
jdbc {
jdbc_driver_library => "/home/vinay/Applications/mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://x.y.w.z:3306/dummyTable" //Connection String
jdbc_user => ""
jdbc_password => ""
statement_filepath => "query.sql"
lowercase_column_names => false
}
}
output {
stdout { codec => json_lines }
elasticsearch {
index => "index1"
document_id => "%{account_id}"
hosts => ["x.x.x.x:9200"] //TEST Server ES IP
}
}
The query statement is :
SELECT
opp.opportunity_id AS opportunity_id,
opp.opportunity_name AS opportunity_name,
acc.account_id AS account_id,
acc.account_name AS account_name,
acc.region AS region,
prd.base_uom AS product_uom,
prd.isGrowth AS is_growth,
prd.isActive AS is_active,
opp.est_revenue AS est_revenue,
opp.opportunity_score AS opportunity_score,
opp.opp_state AS opp_state,
opp.est_close_date AS est_close_date
FROM
opportunity_new AS opp
LEFT JOIN
account1 acc ON opp.account_id = acc.account_id
LEFT JOIN
product_new prd ON opp.product_id = prd.product_id
ORDER BY opportunity_score;
The total number of records using this query is 6658 records in mysql.
But, when i index my data from DEV(mysql) to TEST(elastic search), some records are missing. The count is little less than the original count.
When i try to do the same from DEV(mysql database) to my local ES, the number of records are perfect(same as original).
Please suggest a solution.