# Repeat insert data use logstash-input-jdbc plugin dump data from mysql to elasticsearch

**URL:** <https://discuss.elastic.co/t/repeat-insert-data-use-logstash-input-jdbc-plugin-dump-data-from-mysql-to-elasticsearch/35723>\
**Category:** Logstash\
**Created:** [November 27, 2015, 8:44am UTC](https://discuss.elastic.co/t/repeat-insert-data-use-logstash-input-jdbc-plugin-dump-data-from-mysql-to-elasticsearch/35723 "2015-11-27T08:44:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![506764932](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/506764932/32/22675_2.png) [@506764932](https://discuss.elastic.co/u/506764932)\
**Post date:** [November 27, 2015, 8:44am UTC](https://discuss.elastic.co/t/repeat-insert-data-use-logstash-input-jdbc-plugin-dump-data-from-mysql-to-elasticsearch/35723/1 "2015-11-27T08:44:11Z")

</div>

hi  
i use logstash-input-jdbc plugin dump data from mysql to elasticsearch  
here is my input content inlogstash.conf :

`jdbc { jdbc_connection_string => "jdbc:mysql://192.168.0.49:3306/dfb" jdbc_user => "test" jdbc_password => "test" jdbc_driver_library => "/opt/logstash/mysql-connector-java-5.1.36-bin.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_paging_enabled => "true" jdbc_page_size => "50000" statement_filepath => "jdbc.sql" schedule => "* * * * *" type => "jdbc" }`

the jdbc sql is  
`select h.id as id, h.hotel_name as name, h.photo_url as img, ha.id as haId, ha.finance_person from hotel h LEFT JOIN hotel_account ha on h.id = ha.hotel_id where h.last_modify_time >= :sql_last_start`

when i setting well and first start logstash . it running well. all data is dump to es . and the console is waiting.  
when i execute a insert sql to insert data the console print a message and the data also exist in es.  
but about 30s later . the console print the message again . and es repeat insert the same data again too.  
i test update sql . the same !\*\*

i do not konw what is the matter with logstash . and i have konw idea where the :sql\_last\_start field is worked 。

---

<div class="post-metadata">

**Author:** ![Sasi\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sasi_kumar/32/12988_2.png) [@Sasi\_Kumar](https://discuss.elastic.co/u/Sasi_Kumar)\
**Post date:** [November 5, 2016, 3:54am UTC](https://discuss.elastic.co/t/repeat-insert-data-use-logstash-input-jdbc-plugin-dump-data-from-mysql-to-elasticsearch/35723/2 "2016-11-05T03:54:56Z")

</div>

**Reason:**  
As logstash is a feature that give input for the elasticsearch and its not in-build with elasticsearch and it won't have the knowledge of what data exists in elasticsearch, blindly inserts data into elasticsearch as per the config.

**Solution:**  
Once after completing the full index into the elasticsearch, have a intermediate table that holds the latest updates of the timestamp of table. As per the time stamp you can fire any number of incremental queries and have the scheduler in place. So that incremental query will fetch only the incremental data and scheduler will pick the latest data in a defined time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/repeat-insert-data-use-logstash-input-jdbc-plugin-dump-data-from-mysql-to-elasticsearch/35723/3 "2017-07-06T04:30:57Z")

</div>