Extract from ElasticSearch, into Kafka, continuously add any new ES updates using logstash

Elastic Stack Logstash
1

Hi Team,

My objective is to add latest ES index documents to kafka
Below is my logstash conf ->
'''
input {
elasticsearch {
hosts => ["IP"]
index => "Index_name"
query => '{"query":{"range":{"@timestamp":{"gte": "now-5m"}}}}'
}
}

output {
kafka {
codec => json
topic_id => "topic_name"
bootstrap_servers => "IP"
}
}
'''
it works. But everytime I run it added all the documents to kafka.
It should add only latest updates from index but not all.

Do i need to add date filter, or change the query part? Kindly assist how to do this?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.