# Jdbc input plugin read data multiple times from database **URL:** \ **Category:** Logstash\ **Created:** [June 5, 2019, 9:17am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314 "2019-06-05T09:17:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [June 5, 2019, 9:17am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/1 "2019-06-05T09:17:53Z")
Hi All I am using JDBC input plugin, i have 10,000 records in Database I am setting jdbc\_page\_size as 500 It goes in loop 20, to get all the 10,000 records for this, i schedule this as schedule =\> \*/2 \* \* \* \* With this schedule, every 2 min it will fetch 500 records from database But once the 20 loop is completed, the LS is again going and bringing data from Database so the pipeline is continuously bringing the data, how can we avoid to stop bringing duplicate data once 10,000 records are completed? ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [June 5, 2019, 10:44am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/2 "2019-06-05T10:44:22Z")
The data is being logged to the elasticsearch? If yes you could calculate MD5 hash out of your result and use it as a document ID. All the duplicates will be recorded in the Elasticsearch in the same document ID, and the only \_version will increase. ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [June 5, 2019, 1:39pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/3 "2019-06-05T13:39:08Z")
This is to just avoid duplicate i do not want pipeline to run unnecessary and keep doing indexing and then consuming lot of RAM and CPU by checking documentid. One it loads 10,000 records it should not load data at all ---
**Author:** ![BenjaminD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamind/32/4866_2.png) [@BenjaminD](https://discuss.elastic.co/u/BenjaminD)\ **Post date:** [June 7, 2019, 1:18pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/4 "2019-06-07T13:18:06Z")
Hello. What about your `document_id` in the elasticsearch output settings ? If you don't specify it, each time the request is launched ,it creates new documents. Give it the unique id you have in the database, it should solve your issue. ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [June 7, 2019, 2:13pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/5 "2019-06-07T14:13:35Z")
Hi @BenjaminD I have 2mn records, i am using persistent queue this is what happening with me page0 : 0.5 mn page 1: 0.5 mn (1mn) page2 : 0.5 mn (1.5 mn) page3 : 0.5 mn (2mn) page 4: 0.5 mn ( 2.5 mn) page 5 :0.5mn (3.0 mn) ... ... ... This is going in continuous loop, how can i avoid that. I dont have any problem in using document\_id, i am already aware of that solution, but how can i avoid this continuous loop? I want once 2mn is done it should stop going in second loop and i should be able to schedule it for next day But schedule here works on pagination basis ---
**Author:** ![BenjaminD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamind/32/4866_2.png) [@BenjaminD](https://discuss.elastic.co/u/BenjaminD)\ **Post date:** [June 7, 2019, 2:22pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/6 "2019-06-07T14:22:26Z")
Hi @rkhapre I'm sorry I misunderstood your issue. Sadly, I can't help you as I'm still learning the Elastic Stack. I hope you will find what you're looking for. ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [June 8, 2019, 8:54pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/7 "2019-06-08T20:54:23Z")
Hi , anyone have any idea, how we can solve this Thanks in advance. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2019, 8:59pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-read-data-multiple-times-from-database/184314/8 "2019-07-06T20:59:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.