Send Only updates to Kafka topic

Hi Team,
we need to send the updates received into ElasticSearch Index(example: testcaseindex) to Kafka topic

The Solution i am looking for is:-
To only pick up UPDATES not the whole of index data everytime we schedule and send to kafka topic - below data sends all of data every time we send

input{
elasticsearch{
hosts => "xxxxxxxxxxxx01:9201"
index => "testcaseindex"
size => 1000
scroll => "5m"
docinfo => true
}
}

output{
kafka{
codec => json
topic_id => "POC.CASE.TOPIC"
bootstrap_servers => "XXXXXXXX01.XX.XXXX.com:9092"
}
}

RUN with below
bin/logstash -f logstash-filter.conf

I do not think the elasticsearch input plugin has anything similar to the sql_last_value parameter in the JDBC plugin, so it is likely not able to select only updates according to some criteria.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.