What configuration is used to add newly added data in old indexes?

Elastic Stack Logstash
1

Hello,

Can anyone help me in this?
I have one index made up of database table which is created by using sql query. I want to add newly added data into it. I do not want to index whole data again.
For example: database index has 5 documents. 2 new rows are added into my table from which i have created index. so i want to add that 2 rows in my index which has 5 documents.
What setting i have to do in ES?

Regards,
Priyanka

2

Can anyone help me?

3

Hi Priyanka,
This is not so much a question about elasticsearch but more about whichever system you use to feed it.
Logstash is one example of an ETL tool and here is an example question about using it to do incremental updates from a database.
If you have further questions it's probably best to ask in the Logstash forum

4

Hello,

@Mark_Harwood thanks for reply!! i have marked this question to logstash forum.

can anyone help me to solve following error?
i am using sql_last _value in my logstash config file and try to load incremental data into Elasticsearch from oracle database.

My logstash config file :

input {
  jdbc {
    jdbc_driver_library => "abcd"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "abcde"
    jdbc_user => "test"
	jdbc_password => "xyz"
    jdbc_fetch_size => "10000" 
    statement  => "SELECT * FROM table_name WHERE LASTMODIFIEDDATE > :sql_last_value" 
	last_run_metadata_path => "E:/usr/share/logstash/last_run_metadata/.query.sql"
	use_column_value => true
    tracking_column => "LASTMODIFIEDDATE"
	tracking_column_type =>"timestamp"
 }
 }
 output{
   elasticsearch {
 hosts => "localhost:9200"
  index => "types"
  }
}

and when i am running this i am getting below error:

Error: no implicit conversion of NilClass into String
Exception: TypeError

how to resolve it?

Regards,
Priyanka

5

Is there are stack trace for that error message? Does "E:/usr/share/logstash/last_run_metadata/.query.sql" exist? If so, what does it contain?

6

Hello @Badger,

Yes, '.query.sql' file exist at location E:/usr/share/logstash/last_run_metadata/.query.sql .
And it has timestamp of when logstash config file last run.
for example, --- 2019-07-30 09:13:30.794000000 Z.

Regards,
Priyanka

7

I would try moving that file aside so that it creates the file again.

Again, is there a stack trace associated with that error?

8

Hi @Badger,

No there is not any stack trace associated with that error.

Regards,
Priyanka

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.