X-Pack Watcher: From/Size in webhook input search?

[jay-snee]

I'm setting up Watches with input queries that could potentially return upwards of 10,000 hits.

How would I handle pagination of results in this case?

[spinscale (Alexander Reelsen)]

Hey,

currently the search input of watcher does not support pagination or scroll searches. You could have another process indexing the aggregate of that search into an index, and process this via a watch.

If you explain your use-case a bit, we might be able to find another workaround?

--Alex

[jay-snee]

Hi Alex, thanks for the reply!

It's good to know that it's not supported, at least I know I'm looking for a different solution now.

The use case is that I'm wanting to perform actions on based newly created documents. For the the most part these documents are indexed relatively slowly - usually each watch execution will return less than 15 documents so that's all working fine. However there are times when for a variety of reasons there may well be upwards of a few hundred thousand documents that meet the input search criteria. These documents all need to be acted up too.

Now I'm thinking I might update the watch to just return the number of new documents, and then have the application that receives the webhook notification run a worker task that would pull in and process the results if the payload value was greater than zero. That should hopefully do the trick?

Thanks again!

[spinscale (Alexander Reelsen)]

Hey,

just asking: Your use-case could not be solved by the percolator by any coincidence? This is something that would be applied for every incoming document (on your client side), so you could be sure, not to miss anything.

Another trick might be to have a watch running, that executes your logic for low ingest rates, but triggers something else for high ingestion? You could do that using conditions in your actions.

In addition, you could take your document and combine it with the execute watch API (as an alternative input) and thus fire each document against a watch - not sure if that is what you want, but just a wild thought.

--Alex