JDBC INPUT plugin not syncing all eligible records from postgres db to elasticsearch

Elastic Stack Logstash
1

Hi I have an issue whereby logstash doesn't update all records that are returned by the jdbc-input query to Elasticsearch.As a result we always have to restart logstash to force through the updates.

input plugin config:

jdbc {
jdbc_driver_library => "********"
jdbc_driver_class => "org.postgresql.Driver"
jdbc_connection_string => "${jdbc_connection_string}"
jdbc_user => "${jdbc_user}"
jdbc_password => "${jdbc_password}"
tracking_column => "mi_last_modified_date_unix_ts"
jdbc_paging_enabled => true
jdbc_page_size => 100
use_column_value => true
tracking_column_type => "numeric"
schedule => "${schedule}"
statement => "select *, extract(epoch from to_timestamp(last_modified_date,'YYYY-MM-DD HH24:MI:SS')) as mi_last_modified_date_unix_ts from table where last_modified_date is not null and (extract(epoch from to_timestamp(last_modified_date, 'YYYY-MM-DD HH24:MI:SS')) > :sql_last_value) order by last_modified_date asc"

}

On average there 1500 records that have their last_modified_date updated but we always find that around 100 do not sync through.

Any help would be greatly appreciated.

Thanks.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.