# Duplicate entries when using jdbc pipeline

**URL:** <https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529>\
**Category:** Logstash\
**Tags:** elastic-stack-monitoring\
**Created:** [March 8, 2019, 4:25pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529 "2019-03-08T16:25:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![annk](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@annk](https://discuss.elastic.co/u/annk)\
**Post date:** [March 8, 2019, 4:25pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529/1 "2019-03-08T16:25:21Z")

</div>

Hi,

I am getting duplicate entries when I use my jdbc pipeline below:  
input {  
jdbc{  
jdbc\_driver\_library =\> "/opt/jdbcjars/mariadb-java-client-2.3.0.jar"  
jdbc\_driver\_class =\> "org.mariadb.jdbc.Driver"  
jdbc\_connection\_string =\> "jdbc:mariadb://testhost:3306/otrs"  
jdbc\_validate\_connection =\> true  
jdbc\_user =\> "logstash-user"  
jdbc\_password =\> "logstash-password"  
schedule =\> "\*/5 \* \* \* \*"  
statement =\> "SELECT \* FROM ticket\_info WHERE change\_time \> :sql\_last\_value ORDER BY ticketid ASC"  
use\_column\_value =\> true  
tracking\_column =\> "change\_time"  
tracking\_column\_type =\> "timestamp"  
clean\_run =\> true  
tags =\> ["jdbc","otrsdb"]  
last\_run\_metadata\_path =\> "/opt/jdbc-test/.logstash\_jdbc\_last\_run"  
id =\> "jdbc"  
type =\> "jdbc"  
}  
}  
filter{  
}  
output {  
if [type] == "jdbc" {  
elasticsearch {  
hosts =\> ["[https://elastichost1:1223](https://elastichost1:1223)","[https://elastichost2:1223](https://elastichost2:1223)","[https://elastichost3:1223](https://elastichost3:1223)"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}  
}

I am getting 3 entries in elasticsearch for each database entry.  
document\_id =\> "%{[@metadata][\_id]}"  
But it didn't work and this is what I got

## { "\_index": "logstash-2019.03.07", "\_type": "doc", "\_id": "%{[@metadata][\_id]}", "\_version": 1302, "\_score": null,

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2019, 4:40pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529/2 "2019-03-08T16:40:15Z")

</div>

Have you run logstash with the configuration more than once?

---

<div class="post-metadata">

**Author:** ![annk](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@annk](https://discuss.elastic.co/u/annk)\
**Post date:** [March 8, 2019, 4:53pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529/3 "2019-03-08T16:53:08Z")

</div>

Yes. I am using centralised pipeline management.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2019, 5:19pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529/4 "2019-03-08T17:19:11Z")

</div>

> [@annk](#):
>
> clean\_run =\> true

I believe you should only run with that once.

---

<div class="post-metadata">

**Author:** ![annk](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@annk](https://discuss.elastic.co/u/annk)\
**Post date:** [March 8, 2019, 5:45pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529/5 "2019-03-08T17:45:30Z")

</div>

I tried with that option, But I am still getting 3 entries for every single record read from my db.

I thought the document\_id =\> "%{[@metadata][\_id]}" option will fix it but its now working either.

I am using Elastic 6.3.1

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 5, 2019, 5:45pm UTC](https://discuss.elastic.co/t/duplicate-entries-when-using-jdbc-pipeline/171529/6 "2019-04-05T17:45:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.