Duplication of rows in jdbc logstash input - Using stored procedure

Elastic Stack Logstash
1

I'm fetching data from jdbc input plugin. After running through the whole procedure logstash stops.

Problem is whenever the new rows are added, I run the .conf file and it duplicates the previous data. I have read only access to the database and I call a procedure to fetch the data.

SQL Query - "CALL faredge.transaction_data('2015-01-21 12:30:00','2019-02-11 18:29:00');"

faredge is my schema and transaction_data is my stored procedure name

CONTENTS OF conf file:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://172.23.125.52:3304/faredge"

    jdbc_user => "faredge"
    jdbc_password => "faredge"
	jdbc_driver_library => "C:\Users\giripratap.shekhawat.NUCDOMAIN\Downloads\mysql-connector-java-5.1.47\mysql-connector-java-5.1.47\mysql-connector-java-5.1.47-bin.jar"
    jdbc_driver_class => "Java::com.mysql.jdbc.Driver"
	statement => "CALL `faredge`.`transaction_data`('2015-01-21 12:30:00','2019-02-11 18:29:00');"
}

}
filter {
mutate {
convert => {
"mobile_no" => "string"
}
}
}

output {
stdout {
codec => json
}
elasticsearch {
hosts => ["10.0.51.128:9200"]
index => "payse_new_transac"
retry_on_conflict => 5
}
}

2

Hi,

When you have new rows do you change the query to reflect that? If you would run the query twice it will push the events to elasticsearch again which would look like duplicates.

So, what you want to do it make sure when new rows are added, you only pull those rows in your query.

Paul.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.