So I am noodling different ideas to transform an application that I run. At a high level, I need to be able to support an adhoc query based on a wide range of criteria, and then be able to return back a data set of ID's that will be crammed into a table to be used down stream. So row size is really small, but it could potentially be millions of rows. Currently this is done in a RDBMS and depending on the criteria, and so on it can be quite the sql statement and can be extremely resource intensive.
What I am wondering is this. Could Elasticsearch be used to solve this. Can I use some combo out of this stack to perform an adhoc search (I mean what is an adhoc query other then an adhoc search anyway). The trick is I need to get back the entire result set, if paginated the page size would have to be able to be huge. So if anyone here has done something similar, or has some suggestions on where to look I'm all ears.
--tx