# Incremental data on JDBC

**URL:** <https://discuss.elastic.co/t/incremental-data-on-jdbc/59776>\
**Category:** Logstash\
**Created:** [September 5, 2016, 3:40am UTC](https://discuss.elastic.co/t/incremental-data-on-jdbc/59776 "2016-09-05T03:40:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mormaii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mormaii/32/10790_2.png) [@Mormaii](https://discuss.elastic.co/u/Mormaii)\
**Post date:** [September 5, 2016, 3:40am UTC](https://discuss.elastic.co/t/incremental-data-on-jdbc/59776/1 "2016-09-05T03:40:54Z")

</div>

I'm getting input from the JDBC plugin on logstash like this:

`SELECT epoch,action_id,player_id,world_id,block_id FROM prism_data`

The data looks like this:

```
{
         "epoch" => 1473046041,
     "action_id" => 27,
     "player_id" => 3,
      "world_id" => 3,
      "block_id" => 0,
      "@version" => "1",
    "@timestamp" => "2016-09-05T03:27:21.000Z"
}

```

What I want to do is find a way to check if the data has already been stored on my elasticsearch and if so then don't store it so I don't have duplicate data.

One of my ideas was using the epoch field instead of timestamp so I can get the time of the event instead of when the data was added.

Another idea was to truncate the table after getting the info from it so we ensure that all data stored will not be duplicated but I don't think that's the best way possible.

edit: The whole point of this is to get incremental data from a table so I have data synchronized with my elastic.

---

<div class="post-metadata">

**Author:** ![Mormaii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mormaii/32/10790_2.png) [@Mormaii](https://discuss.elastic.co/u/Mormaii)\
**Post date:** [September 5, 2016, 5:39am UTC](https://discuss.elastic.co/t/incremental-data-on-jdbc/59776/2 "2016-09-05T05:39:56Z")

</div>

Solved it myself like this:

```
statement => "SELECT epoch,action_id,player_id,world_id,block_id FROM prism_data WHERE epoch >:sql_last_value"
use_column_value => true
tracking_column => epoch

```

"sql\_last\_value" stores the last value from the previous run so it will only add new data if the epoch is greater than last run. This prevents duplicates and guarantees incremental data.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:39am UTC](https://discuss.elastic.co/t/incremental-data-on-jdbc/59776/3 "2017-07-06T04:39:58Z")

</div>