# Incremental data

**URL:** <https://discuss.elastic.co/t/incremental-data/223688>\
**Category:** Logstash\
**Created:** [March 16, 2020, 5:31am UTC](https://discuss.elastic.co/t/incremental-data/223688 "2020-03-16T05:31:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chelseia](https://avatars.discourse-cdn.com/v4/letter/c/d78d45/32.png) [@chelseia](https://discuss.elastic.co/u/chelseia)\
**Post date:** [March 16, 2020, 5:31am UTC](https://discuss.elastic.co/t/incremental-data/223688/1 "2020-03-16T05:31:10Z")

</div>

I have some records already existed in table.I have modified some records after pushed into the table.but the modified data was not reflected in elastic search.how to achieve this?

---

<div class="post-metadata">

**Author:** ![rameshkr1994](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshkr1994/32/59029_2.png) [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\
**Post date:** [March 18, 2020, 1:27pm UTC](https://discuss.elastic.co/t/incremental-data/223688/2 "2020-03-18T13:27:04Z")

</div>

Hi @chelseia.

`Please use below config file for logstash to get incremental update in existing elastic search index.`

```auto
input {
  jdbc {
    jdbc_driver_library => "C:\Users\XXX\Downloads\Elasticsearch\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
    #jdbc_driver_class => "com.microsoft.jdbc.sqlserver.SQLServerDriver"
	jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver:XXXXX;integratedSecurity=false;"
	
    
    jdbc_user => "USER"
	jdbc_password => "PASSWORD"
	jdbc_validate_connection => true
	tracking_column => "dateinserted"
	use_column_value =>true

    statement => "SELECT * FROM COURSE where dateinserted >:sql_last_value;"
	last_run_metadata_path => "C:\Users\Ramesh.kumar\.logstash_jdbc_last_run"
	#schedule => "* * * * * *"
  }
}

output {
  elasticsearch {
    document_id => "%{id}"
	document_type => "_doc"
    index => "dbdataindex_testauto"
	hosts => ["localhost:9200"]
	}
	stdout{
  codec => rubydebug
	
  }
}

```

> Note-please modify the logstash.config as per your information.  
> un-comment #schedule =\> "\* \* \* \* \* \*" if you want to provide schedule time

Thanks  
HadoopHelp

---

<div class="post-metadata">

**Author:** ![chelseia](https://avatars.discourse-cdn.com/v4/letter/c/d78d45/32.png) [@chelseia](https://discuss.elastic.co/u/chelseia)\
**Post date:** [April 1, 2020, 2:34pm UTC](https://discuss.elastic.co/t/incremental-data/223688/3 "2020-04-01T14:34:40Z")

</div>

Thank you.but I dont have any date column to use as the sql\_last\_value.and my table consist of neither timestamp nor incremental id.so how to achieve the real time data using logstash?

---

<div class="post-metadata">

**Author:** ![rameshkr1994](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshkr1994/32/59029_2.png) [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\
**Post date:** [April 2, 2020, 9:04am UTC](https://discuss.elastic.co/t/incremental-data/223688/4 "2020-04-02T09:04:20Z")

</div>

Hi @chelseia.

you can use separate row0num or create separate row-num columns= in sql query and store into Elasticsearch as index then second time trace this column - row-num?

> you will get unique records /every time your scheduler will get newly data and updated data ?

Thanks  
HadoopHelp

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2020, 9:04am UTC](https://discuss.elastic.co/t/incremental-data/223688/5 "2020-04-30T09:04:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.