Incremental Refresh

Hello Everyone,
I am new to ELK Stack.

I have data in postgresql which i need to do some visualizations in Kibana.
Steps i have followed

Creating a logstash config file which reads data from postgresql and pushes it in elasticsearch, below is the config file.

input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://localhost:5432/dvdrental"
jdbc_user => "postgres"
jdbc_password => "123456"
jdbc_validate_connection => true
jdbc_driver_library => "C:\kabana\logstash-5.1.2\logstash-5.1.2\drivers\postgresql-9.4-1200-jdbc41.jar"
jdbc_driver_class => "org.postgresql.Driver"
statement => "SELECT * from public.address"
output {
elasticsearch {
#protocol => http
index => "address_table"
document_type => "address"
document_id => "%{address_id}"
hosts => ["localhost:9200"]

i am successful in loading data from PostgreSQL to elastic search.
but what i want is an incremental refresh, such that when new data is loaded in the PostgreSQL table only the new data should be added in elasticsearch through logstash and the data loading should happen every hour.(Like appending the new data to existing data in elasticsearch)
Please help me with the same.

Have you read what's said about sql_last_value in the jdbc input documentation?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.