Incremental data

Elastic Stack Logstash
1

I have some records already existed in table.I have modified some records after pushed into the table.but the modified data was not reflected in elastic search.how to achieve this?

2

Hi @chelseia.

Please use below config file for logstash to get incremental update in existing elastic search index.

input {
  jdbc {
    jdbc_driver_library => "C:\Users\XXX\Downloads\Elasticsearch\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
    #jdbc_driver_class => "com.microsoft.jdbc.sqlserver.SQLServerDriver"
	jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver:XXXXX;integratedSecurity=false;"
	
    
    jdbc_user => "USER"
	jdbc_password => "PASSWORD"
	jdbc_validate_connection => true
	tracking_column => "dateinserted"
	use_column_value =>true

    statement => "SELECT * FROM COURSE where dateinserted >:sql_last_value;"
	last_run_metadata_path => "C:\Users\Ramesh.kumar\.logstash_jdbc_last_run"
	#schedule => "* * * * * *"
  }
}

output {
  elasticsearch {
    document_id => "%{id}"
	document_type => "_doc"
    index => "dbdataindex_testauto"
	hosts => ["localhost:9200"]
	}
	stdout{
  codec => rubydebug
	
  }
}

Note-please modify the logstash.config as per your information.
un-comment #schedule => "* * * * * *" if you want to provide schedule time

Thanks
HadoopHelp

3

Thank you.but I dont have any date column to use as the sql_last_value.and my table consist of neither timestamp nor incremental id.so how to achieve the real time data using logstash?

4

Hi @chelseia.

you can use separate row0num or create separate row-num columns= in sql query and store into Elasticsearch as index then second time trace this column - row-num?

you will get unique records /every time your scheduler will get newly data and updated data ?

Thanks
HadoopHelp

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.