Logstash hangs on using document_id while creating index for first time

Elastic Stack Logstash
1 
input {
      jdbc {
    jdbc_driver_library => "sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=transactions_sample_db"
    jdbc_user => "sa"
    jdbc_password => "awesome"
    statement => "SELECT * from sample"
    # schedule => "* * * * *"
      }
    }
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["localhost:9200"]
index => "test_index_4"
document_id => "%{[_id]}"
}
}

the above code describes my config file,
and am doing all configurations and everything on my windows laptop (4gb ram) , am new to logstash.
The Problem is:

  1. If i run logstash to create a new index that id not in elasticsearch without document_id in conf file it works fine, but if i create a new index with document_id it hangs forever, here are all files i have in logstash attached
  2. The document_id works fine if index is already present, then it avoids duplicates.
  3. Do i have to change my config file when am creating index for first time, and then add document it later? as _id is not present for first time?

(my elastic search is always running and also i can see indexes on http://localhost:9200/_cat/indices?v)

image
image.png1366×768 59 KB

image
image.png1366×768 212 KB

image
image

2

Does the event really have an _id field? From the log it seems the correct name is t_id.

3

for the first time upload it doesnt has.
i tried using t_id but that doesn't help me avoid duplicacy then

image
image.png1366×768 95.3 KB

image
image.png1366×768 80.7 KB

4

If the _id field does not exist, you will end up updating the same document repeatedly, which will be very inefficient and slow as this will result in a lot of very small refreshes (which are expensive) in recent version.

5

I got it but can you please help me know what to use in place of _id? t_id and id are my primary keys in database, I tried using t_id , it duplicated them, i had 30 lac documents and running it using t_id added 20k more again

6

thanks i changed them and it worked fine!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.