# Logstash get same data from database **URL:** \ **Category:** Logstash\ **Created:** [November 28, 2018, 11:45am UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555 "2018-11-28T11:45:58Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ali1](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@ali1](https://discuss.elastic.co/u/ali1)\ **Post date:** [November 28, 2018, 11:45am UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/1 "2018-11-28T11:45:58Z")
Hello , im using logstash to get data from my database , and all forks fine , but when i change the number of line in my table for exemple i delete some rows , i still find the same rows as document in my elasticsearch. **my example** : i have 10 rows in my table , i run logstash and i find my 10 rows as document in ES , i go back to my database i delete 5 rows i run logstash i go back to ES i fin 10 documents. please do you have any idea or config to inform logstash to get the same line as the table and delete the existing documents if doesn't exists in the select query. Regards ! ---
**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\ **Post date:** [November 28, 2018, 11:50am UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/2 "2018-11-28T11:50:53Z")
You will need to show us some configuration in order to help ---
**Author:** ![ali1](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@ali1](https://discuss.elastic.co/u/ali1)\ **Post date:** [November 28, 2018, 11:58am UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/3 "2018-11-28T11:58:16Z")
Hi this my pipline config : ``` input { jdbc { jdbc_connection_string => "dburl..." jdbc_user => "username" jdbc_password => "password" jdbc_driver_library => "/opt/elasticsearch/logstash/drivers/ojdbc7.jar" jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" schedule => "*/1 * * * *" statement => "select * from my_table" } } filter { mutate { rename => { "numfolder" => "numFolder" "creationdate" => "creationDate" "stateuser" => "stateUser" "isvalid" => "isValid" "isbroken" => "isBroken" } convert => { "isValid" => "boolean" "isBroken" => "boolean" } } } output { elasticsearch { index => "users" document_type => "infos" document_id => "%{no_info}" hosts => ["localhost:9200"] } } ``` Thanks ---
**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\ **Post date:** [November 28, 2018, 12:01pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/4 "2018-11-28T12:01:09Z")
Doesn't look like you are running your JDBC on schedule, you are only doing one request? Also you haven't deleted old data from ES after dropping statements, as far as I can tell? ---
**Author:** ![ali1](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@ali1](https://discuss.elastic.co/u/ali1)\ **Post date:** [November 28, 2018, 12:07pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/5 "2018-11-28T12:07:10Z")
sorry i forget to add the line `schedule => "*/1 * * * *"` yes im using schedule property to get data from database every minute, my issue is how to inform ES to drop old data dynamically because dropping statements in database are trigred by users so i can't delete data from ES manually ---
**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\ **Post date:** [November 28, 2018, 12:10pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/6 "2018-11-28T12:10:14Z")
If you only want the current data and no historic, you could use the "action =\> update" setting in the output. You will need to define a unique ID to update against, but since this is a database it shouldn't be an issue! ---
**Author:** ![ali1](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@ali1](https://discuss.elastic.co/u/ali1)\ **Post date:** [November 28, 2018, 12:21pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/7 "2018-11-28T12:21:54Z")
you mean when using input as database "action =\> update" is set by default ? ---
**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\ **Post date:** [November 28, 2018, 12:45pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/8 "2018-11-28T12:45:00Z")
Not according to my docs: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/2/d24a63eb134a9b9c6da647c9e794d720664fb6bf.png) I have an idea how to solve it, testing it now. ---
**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\ **Post date:** [November 28, 2018, 3:02pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/9 "2018-11-28T15:02:13Z")
I figured out a great solution for this, simply create a second input filter using elasticsearch as the input to pull all records from your index older than 2 minutes old. Since you are updating all records every minute this should be suitable. Add a tag for any returned results. On your output, create a secondary output for that tag and again using elasticsearch as the output, using the delete action this time. I've tested this and confirmed it 100% works. Took some effort to think of a solution but it works well. ---
**Author:** ![ali1](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@ali1](https://discuss.elastic.co/u/ali1)\ **Post date:** [November 28, 2018, 3:35pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/10 "2018-11-28T15:35:54Z")
thank you for your time can you please share with me your configuration file. regards ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 26, 2018, 3:43pm UTC](https://discuss.elastic.co/t/logstash-get-same-data-from-database/158555/11 "2018-12-26T15:43:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.