Having the http_poller input be aware of how to paginate is something we've discussed internally. I personally feel it's not something we can achieve because of the ways that things present pagination, such as:
- Fetch the main page, get a token that you can query repeatedly in the future to get the next page of results (Elasticsearch Scroll API does this)
- Fetch the first page, and the response includes an identifier for the next page (Github API does this, as does your Microsoft service, from your description)
- Fetch pages by number (1, 2, 3, 4) and stop fetching when the current page has no results.
There are probably more pagination strategies, and I'm not sure we can prepare the http_poller plugin to support them all.
In cases such as this, I generally recommend that a new input plugin be created that has specific knowledge of how to read data from a specific data source. In this case, it would be my recommendation to have a custom input plugin that knows how to handle the pagination strategy deployed by the Microsoft Table Service.