# Updating data which are newly added

**URL:** <https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236>\
**Category:** Logstash\
**Created:** [March 28, 2017, 5:14am UTC](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236 "2017-03-28T05:14:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Varun\_Patel](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Varun\_Patel](https://discuss.elastic.co/u/Varun_Patel)\
**Post date:** [March 28, 2017, 5:14am UTC](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236/1 "2017-03-28T05:14:14Z")

</div>

Hi,  
I am using jdbc connection for fetching data from database.

I have single index. Let say Contacts.

I am able to fetch data from database. But what i want is when i run logstash for second time it should only add data to elasticsearch which are new in database.

Explanation:

1. Running logstash for first time  
ES will have 5 data in 'Contacts' index. Database has 5 data
2. Running logstash for second time  
ES will have 11 data in 'Contacts' index. Database has 6 data

So what is happening is i am having multiple entries of same data. I want only newly added data when running logstash second tme.

How to achieve this ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 28, 2017, 7:40am UTC](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236/2 "2017-03-28T07:40:19Z")

</div>

This has been discussed several times before, but the idea is to not use Elasticsearch's automatically generated document id but set your own document id based on one or more fields that originate from columns returned from the database query.

The elasticsearch output's `document_id` option can be used for this, and if your events for example get an `id` field with (typically) a primary key from the database you can use `document_id => "%{id}"` to use that id as the document id in ES.

---

<div class="post-metadata">

**Author:** ![Varun\_Patel](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Varun\_Patel](https://discuss.elastic.co/u/Varun_Patel)\
**Post date:** [March 28, 2017, 10:23am UTC](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236/3 "2017-03-28T10:23:49Z")

</div>

Thanks @magnusbaeck.

Got more cleared information form the below post

> [@Logstash adding duplicate rows for every run](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-for-every-run/44775):
>
> Hello World, We implemented a solution to push Microsoft SQL Data in form of simple rows from SQL Server to Logstash which will index the data to Elasticsearch. The data (SQL rows and columns) are getting successfully by using a simple jdbc plugin to logstash and logstash indexing the data to elasticsearch. But, everytime we try to pump the data (same data) what its doing is adding duplicate records with same data but "with different \_id". SO everytime we pump the data from our client to logsta…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 25, 2017, 10:23am UTC](https://discuss.elastic.co/t/updating-data-which-are-newly-added/80236/4 "2017-04-25T10:23:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.