# SQL to ElasticSearch, only new rows?

**URL:** <https://discuss.elastic.co/t/sql-to-elasticsearch-only-new-rows/261476>\
**Category:** Elasticsearch\
**Created:** [January 18, 2021, 9:21pm UTC](https://discuss.elastic.co/t/sql-to-elasticsearch-only-new-rows/261476 "2021-01-18T21:21:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sath/32/82475_2.png) [@sath](https://discuss.elastic.co/u/sath)\
**Post date:** [January 18, 2021, 9:21pm UTC](https://discuss.elastic.co/t/sql-to-elasticsearch-only-new-rows/261476/1 "2021-01-18T21:21:20Z")

</div>

Hello,

I'm relatively new to the ELK stack. I'm wondering how I could send data from a SQL table to Elasticsearch in a way that every time it runs, it only sends the rows that are new since the last time it ran. The SQL table contains Horizon View event logs.

Currently, for testing, I have setup JDBC connection which successfully gathers the content of the SQL table and sends it via Logstash to Elasticsearch. The problem with this is, it sends the entire table every time. So I'm looking for an Elasticsearch solution that either monitors the SQL table and sends information to Elasticsearch as rows get added to the table, or a way to prevent Logstash from sending the entire table every time, so I could even setup a scheduled transfer.  
Maybe with beats?

Thanks,

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [January 18, 2021, 9:42pm UTC](https://discuss.elastic.co/t/sql-to-elasticsearch-only-new-rows/261476/2 "2021-01-18T21:42:09Z")

</div>

Check out the `tracking_column` and `use_column_value` [parameters](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-tracking_column).

If you have a DB with an incrementing field you can use that to do this. Say you have a field for `id` and it increments with new values for new data. The first time you run this it will grab all the data and then save the `id` on the local file system of the last one ran. Then when you run it again you would do where that `id > last_value`. [See example](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_prepared_statements).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 18, 2021, 9:43pm UTC](https://discuss.elastic.co/t/sql-to-elasticsearch-only-new-rows/261476/3 "2021-01-18T21:43:28Z")

</div>

I shared most of my thoughts there: [http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/](http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/)

Basically, I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.

Have also a look at this ["live coding" recording](https://www.elastic.co/fr/blog/how-to-add-powerful-search-existing-sql-applications-elasticsearch-video-tutorial).

But this supposes you have access to the application which writes the data to the database.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 15, 2021, 9:43pm UTC](https://discuss.elastic.co/t/sql-to-elasticsearch-only-new-rows/261476/4 "2021-02-15T21:43:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.