RESTful APIs and http_poller input plugin

I've found myself in a few situations where there is a RESTful API that I want to consume and which updates over time with newer entries.

I wonder what's the best way to be periodically probing the API and consistently getting all the records without missing any of them?

I realize http_poller is stateless. Would the safest way be to always be querying a larger block of entries and deduplicating them through unique identifiers?

Alternatively, what I've been doing so far is write API apps that consume the API and output a log file that I can then use Filebeat to send to Logstash.

Wonder what are the best practices when it comes to RESTful APIs.

Update: Since I first wrote this post a couple of months ago, I've resorted into client-side scripting and have written an updated post with my more detailed thoughts on the pros and cons between client- and Logstash- side coding, see: REST API, http_poller and ruby code

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.