I've found myself in a few situations where there is a RESTful API that I want to consume and which updates over time with newer entries.
I wonder what's the best way to be periodically probing the API and consistently getting all the records without missing any of them?
I realize http_poller is stateless. Would the safest way be to always be querying a larger block of entries and deduplicating them through unique identifiers?
Alternatively, what I've been doing so far is write API apps that consume the API and output a log file that I can then use Filebeat to send to Logstash.
Wonder what are the best practices when it comes to RESTful APIs.
Update: Since I first wrote this post a couple of months ago, I've resorted into client-side scripting and have written an updated post with my more detailed thoughts on the pros and cons between client- and Logstash- side coding, see: REST API, http_poller and ruby code