I am working on a project where we will be modeling 1-n relations with
potentially hundreds of thousands of endpoints for a single
We are currently storing the data in Riak, and this seems to work
fine. But the relations are of course a problem.
Riak links are implemented in a way that limits the number of links on
a single object to few thousands, so I figured I could use Riak Search
to search for relationship documents of the form "user_id: ...,
However, it seems that Riak Search scales pretty badly when you get
hundreds of thousands of hits. Example: Counting the number of hits
when they are 100.000 takes 10 times as longs as counting the number
of hits when they are 10.000. Not too surprising, perhaps, but it
makes the count take several seconds for big relations.
So, I am thinking, can I do this with elasticsearch, or some other
lucene based engine?
This makes the question I would need an answer to this: Is counting
hits for a search having hundreds of thousands of hits in
elasticsearch a costly operation? Does it scale as linearly as in Riak