Hi Team,
I'm working on an incremental data pipeline using Logstash. I'm using the Elasticsearch input plugin to retrieve the last processed document from an index. I intend to store the relevant field from this document in a variable and use it in the subsequent JDBC input to fetch incremental data.
However, I'm currently encountering an issue where the Elasticsearch input is returning all documents instead of the single expected document.
input {
elasticsearch {
hosts => ["https://es01:9200"]
index => "agg-log*"
query => '{
"_source": false,
"query": {
"match_all": {}
},
"from": 0,
"size": 1,
"sort": [
{
"PaymentResponseReceivedAtPublic_M5": {
"order": "desc"
}
}
]
}'
user => "elastic"
password => "XXXXXX"
ssl_enabled => true
ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca/ca.crt"
response_type => 'hits'
docinfo => true
docinfo_target => "[@metadata][doc]"
add_field => {
identifier => "%{[@metadata][doc][_id]}"
}
search_api => 'search_after'
size => 1
schedule => "*/3 * * * *"
}
}
filter {
ruby {
code => "
require 'json'
logger.info('Full Elasticsearch Response: ' + event.to_json)
"
}
}
I've already set the size
parameter to 1 within the Elasticsearch query, but it's not limiting the results as expected.
Could you please help me identify the cause of this issue and suggest the necessary modifications to the configuration to retrieve only the single desired document?