Adding extra data

Elastic Stack Logstash
1

Help me! Why my logstash with this config file adding the same data repeatedly

input {
jdbc {
jdbc_connection_string => "connection_string"
jdbc_user => "user"
jdbc_password => "password"
jdbc_driver_library => "driver_library"
jdbc_driver_class => "org.postgresql.Driver"
statement => 'query'
}
}
filter {

}

output {
elasticsearch {
hosts => "http://localhost:9200"
index => index
document_id => "%{[ID]}"
document_type => doc
user => elastic
password => password
}
}

2

Imagine I asked you the same question, would you be able to give an accurate answer with the amount of information provided? Please revise your question.

3

Because the query returns the same data repeatedly? Have you looked at what the jdbc input documentation says about state and the sql_last_value parameter?

4

My logstash with this config file adding the same data repeatedly. It is like: I have 100 rows in database, but with a few time I will have 100...200...300...etc Why logstash adding the same data again and again?

5

I saw this parameter, but it dont help my problem

6

You've configured the elasticsearch output to use the contents of the ID column as the document id. If that works as intended I don't see how you can possibly have 200 documents in ES. What document ids do you have, and how do they relate to the database table's ID column?

7

If I have this row document_id => "%{[ID]}", then I have updating my data all time.
If I dont have this row document_id => "%{[ID]}", then I have adding same data in elastic.

8

You're asking the same question as a couple of days ago (Updating data). Please continue that thread instead. I didn't see a response to my last post ("If your table has a column ...").

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.