The existing Apache Storm EsSpout is designed to use a scroll to search through ES once and then exit. It appears to do this very well and quite reliably.
However, in my application I'd like to be able to repeatedly run the query and streams the results. If there are no documents the Spout should sit idle for a configurable amount of time before checking again.
Does anyone have any advice they can share on extending EsSpout for this functionality? Before diving in I wanted to ask if anyone had explore this extension before and where the pitfalls may lie. Thanks.