Logstash missing records in elastic search


(Vinay Garg) #1

Hi,
I have used the configration for logstash to dump data from MYSQL 5.5 to Elastic Search 6.1.2.
input {
jdbc {
jdbc_driver_library => "/home/ubuntu/mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://IP:PORT/DATABASE"
jdbc_user => "username"
jdbc_password => "password"
statement_filepath => "/queries/abc.sql"
lowercase_column_names => false
}
}
output {
stdout { codec => json_lines }
elasticsearch {
index => "accounts"
document_type => "account"
document_id => "%{id}"
hosts => ["elastic_ip:9200"]
}
}

When i run the logstash on my local machine(input database present on a Test server and ES on local machine), the results are accurate.
But when I try to run logstash on Test Server( both database and ES on Test Server), some records are missing randomly.

I am stuck at this point. Please help me out.
For other indexes which have same configration with more than 50000 records are ingested correctly, but account index is missing the documents.


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.