Hello Everyone,
I am new to ELK Stack.
I have data in postgresql which i need to do some visualizations in Kibana.
Steps i have followed
Creating a logstash config file which reads data from postgresql and pushes it in elasticsearch, below is the config file.
input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://localhost:5432/dvdrental"
jdbc_user => "postgres"
jdbc_password => "123456"
jdbc_validate_connection => true
jdbc_driver_library => "C:\kabana\logstash-5.1.2\logstash-5.1.2\drivers\postgresql-9.4-1200-jdbc41.jar"
jdbc_driver_class => "org.postgresql.Driver"
statement => "SELECT * from public.address"
}
}
output {
elasticsearch {
#protocol => http
index => "address_table"
document_type => "address"
document_id => "%{address_id}"
hosts => ["localhost:9200"]
}
}
i am successful in loading data from PostgreSQL to elastic search.
but what i want is an incremental refresh, such that when new data is loaded in the PostgreSQL table only the new data should be added in elasticsearch through logstash and the data loading should happen every hour.(Like appending the new data to existing data in elasticsearch)
Please help me with the same.