# JDBC logging duplicate row

**URL:** <https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297>\
**Category:** Logstash\
**Created:** [November 16, 2021, 10:16am UTC](https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297 "2021-11-16T10:16:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![LinhNT1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/linhnt1/32/79658_2.png) [@LinhNT1](https://discuss.elastic.co/u/LinhNT1)\
**Post date:** [November 16, 2021, 10:16am UTC](https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297/1 "2021-11-16T10:16:07Z")

</div>

Hi there,  
I have a logtash with JDBC input like below

```auto
jdbc {
        jdbc_driver_library => "${LOGSTASH_JDBC_DRIVER_JAR_LOCATION}"
        jdbc_driver_class => "${LOGSTASH_JDBC_DRIVER}"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        jdbc_paging_enabled => true
        jdbc_page_size => 1000
        tracking_column => "unix_ts_in_secs"
        use_column_value => true
        last_run_metadata_path => "/usr/share/logstash/jdbc-last-run/.last_run"
        tracking_column_type => "numeric"
        schedule => "*/5 * * * * *"
        statement => "
            SELECT *, UNIX_TIMESTAMP(updated_at) AS unix_ts_in_secs FROM tbl WHERE updated_at > FROM_UNIXTIME(:sql_last_value) AND updated_at<= now() ORDER BY updated_at, id
          "
    }

```

Example: MySQL database have 1 row pass the condition of above sql statement, but the log print many row (2 or more) with the same data (duplicate), only @timestamp field is different.

 ![Screen Shot 2021-11-16 at 17.08.05](https://us1.discourse-cdn.com/elastic/original/3X/1/1/1180f923c913ba10347f869601474a25704a4335.png)

Finally, only 1 row inserted to Elasticsearch, but I don't know why Logstash print many duplicate data (only @timestamp different). Is it affect performance or make high CPU/Memory usage?

Thank you!

---

<div class="post-metadata">

**Author:** ![Alex\_Marquardt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_marquardt/32/42925_2.png) [@Alex\_Marquardt](https://discuss.elastic.co/u/Alex_Marquardt)\
**Post date:** [November 16, 2021, 7:42pm UTC](https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297/2 "2021-11-16T19:42:22Z")

</div>

> [@LinhNT1](#):
>
> Finally, only 1 row inserted to Elasticsearch, but I don't know why Logstash print many duplicate data (only @timestamp different). Is it affect performance or make high CPU/Memory usage?
> 
> Thank you!

You could specify a `document_id` in the output of your pipeline to ensure that multiple copies of the same document overwrite a single document in Elasticsearch. You can find an example of this in the output of the example pipeline given in: [How to keep Elasticsearch synced with a RDBMS using Logstash and JDBC | Elastic Blog](https://www.elastic.co/blog/how-to-keep-elasticsearch-synchronized-with-a-relational-database-using-logstash).

---

<div class="post-metadata">

**Author:** ![LinhNT1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/linhnt1/32/79658_2.png) [@LinhNT1](https://discuss.elastic.co/u/LinhNT1)\
**Post date:** [November 17, 2021, 1:02am UTC](https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297/3 "2021-11-17T01:02:46Z")

</div>

Thank you, as I said, only one row to be insert/update to Elasticsearch (document\_id is specified in output). My problem is I don't know why Logstash logging a jdbc input row many time with same data (only @timestamp different). Can you help me explain it?

---

<div class="post-metadata">

**Author:** ![LinhNT1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/linhnt1/32/79658_2.png) [@LinhNT1](https://discuss.elastic.co/u/LinhNT1)\
**Post date:** [November 22, 2021, 8:46am UTC](https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297/4 "2021-11-22T08:46:02Z")

</div>

Hi, finding anyone can help me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2021, 8:46am UTC](https://discuss.elastic.co/t/jdbc-logging-duplicate-row/289297/5 "2021-12-20T08:46:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.