I've millions of records in Elasticsearch and sending outputs as individual files using file output plugin.
I've noticed logstash extraction rate is high and used a sleep plugin to slow down after every few secs or every nth event.
Is there any better way to configure logstash to pull elastic data only after certain duration? Also, if the pipeline is failed for some reason, logstash is executing all the way from the beginning. Is there a way to control with an off-set point so logstash can continue from the breaking point?