Elastic as an adhoc query engine

So I am noodling different ideas to transform an application that I run. At a high level, I need to be able to support an adhoc query based on a wide range of criteria, and then be able to return back a data set of ID's that will be crammed into a table to be used down stream. So row size is really small, but it could potentially be millions of rows. Currently this is done in a RDBMS and depending on the criteria, and so on it can be quite the sql statement and can be extremely resource intensive.

What I am wondering is this. Could Elasticsearch be used to solve this. Can I use some combo out of this stack to perform an adhoc search (I mean what is an adhoc query other then an adhoc search anyway). The trick is I need to get back the entire result set, if paginated the page size would have to be able to be huge. So if anyone here has done something similar, or has some suggestions on where to look I'm all ears.

Have you taken a look at scroll searches, a point in time snapshot that you can paginate through? Would that work in this case?


Scroll does look like it might do what I need. I'm just wondering if anyone here has done something along the lines of what I would like to do. I'd rather not go do a journey of discovery off the edge of the world :smiley:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.