# Synch pgsql with elastic with jdbc plugin

**URL:** <https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665>\
**Category:** Logstash\
**Created:** [October 22, 2019, 2:19pm UTC](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665 "2019-10-22T14:19:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![skowron-line](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/skowron-line/32/43528_2.png) [@skowron-line](https://discuss.elastic.co/u/skowron-line)\
**Post date:** [October 22, 2019, 2:19pm UTC](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665/1 "2019-10-22T14:19:21Z")

</div>

Hi I having trouble with syncing data from db to elastic, the problem is that when Iam loading data, sometimes Im missing 3, 4 records sometimes more in my elastic index  
I have 10 workers which load data to database.  
My logstash pipeline

```
input {
    jdbc {
        jdbc_driver_library => "/usr/share/logstash/postgresql.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_connection_string => "jdbc:postgresql://pgsql/broker"
        jdbc_user => "user"
        jdbc_password => "password"
        jdbc_paging_enabled => false
        jdbc_page_size => 100
        jdbc_validate_connection => true
        statement => "SELECT CAST(metadata AS text), pwp_id, updated_at::TEXT FROM pwps WHERE updated_at > :sql_last_value ORDER BY updated_at"
        use_column_value => true
        tracking_column => "updated_at"
        tracking_column_type => "timestamp"
        schedule => "*/2 * * * * *"
        record_last_run => true
        last_run_metadata_path => "/usr/share/logstash/.sync_pwp_last_run"
    }
}
filter {
    json {
        source => metadata
        remove_field => ["metadata"]
    }
}
output {
    elasticsearch {
        index => "index"
        hosts => ["http://elasticsearch:9200"]
        document_id => "%{id}"
    }
    stdout {
        codec => rubydebug
    }
}

```

and my table schema is

```
CREATE TABLE "public"."pwps" (
    "id" uuid NOT NULL,
    "pwp_id" character varying(100),
    "metadata" jsonb,
    "updated_at" timestamp(6) DEFAULT now(),
    "pwp_type_id" integer,
    CONSTRAINT "pwps_pkey" PRIMARY KEY ("id")
) WITH (oids = false);

```

Is threre anyone how had similar problem? And solved it?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [October 22, 2019, 2:24pm UTC](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665/2 "2019-10-22T14:24:59Z")

</div>

Is id unique in SQL database ? if yes I guess the issue is with the index seeting parameter refresh\_interval, the document is not immdiately available for search for any update by an other logstash worker if id is duplicated.  
Reference : [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html)

You may try to enforme more unicity by combining id and updated\_at to get a unique document\_id

Hope this can help

---

<div class="post-metadata">

**Author:** ![skowron-line](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/skowron-line/32/43528_2.png) [@skowron-line](https://discuss.elastic.co/u/skowron-line)\
**Post date:** [October 22, 2019, 2:47pm UTC](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665/3 "2019-10-22T14:47:00Z")

</div>

The id is unique, but I use upstream to override existing document.  
The problem is that missing documents are never available in index, so I asume that they are never pushed to index

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 19, 2019, 2:47pm UTC](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665/4 "2019-11-19T14:47:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.