Hi,
I am fetching data from mssql database table which is keep on updating So I want to fetch only the new entries from the table but each time It is fetching all the data.
like if there are total 5 records then after 1 min records are increasing to 10 then 15 and so on . my script is as below.
input {
#demo
jdbc {
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://XXXXXXX:1433;databaseName=XXXXXXXX"
jdbc_user => "XXXXXX"
jdbc_password => "XXXXXX"
schedule => "* * * * *"
use_column_value => true
tracking_column => date
tracking_column_type => "timestamp"
statement=>"select id,date,tripno from(select id,date(timestamp) as date,tripno from table group by id)a where date>=:sql_last_value"
type => "tabledata"
}
}
output {
elasticsearch {
hosts => "https://localhost:9200"
user => "XXXX"
password => "XXXXX"
index => "%{type}"
#action => "update"
ssl => false
ssl_certificate_verification =>false
# cacert =>"/home/azureuser/cert/root-ca.pem"
ilm_enabled => false
}
#stdout { codec => rubydebug }
}
please Tell me where I am doing it wrong .every min index size is getting increased. do i need to apply fingerprint or if can edit property here in the same script .Thanks In advance