Logstash-elasticsearch input plugin, size not working as expeted


(Gaurav Singhal) #1

Problem: Every 20 minutes logstash fetches all the records matching as per query in batches of 100(size=100) and waits for next schedule.

in input plugin i have something like this:

elasticsearch{
size => 100
schedule => "*\20 * * * *"
}

NEED: I wanted to retrieve only 100 records from matching set of records against the query every 20 minutes from elasticsearch using logstash.

I will really appreciate the some help here.


(Gaurav Singhal) #2

I have changed my logstash configuration and removed size parameter.
Now it is like this:

input {
elasticsearch{
query => '{
"size": 5,
"sort": [
{
"esUpdateDate": {
"order": "desc"
}
}
],
"_source": [
"esUpdateDate",
"beUpdateDate"
]
}
schedule => "*\20 * * * *"
}
}

This also not restricting the size of documents returned by the query.

Really appreciate some help here.