Hi,
I am looking for options in elasticsearch to design an index(s) that can allow personalized searches.
Lets assume the example given in ES 6.8 documentation on searching twitter followers. In this example it suggest updating the user document's follower's field each time someone new follows them. So when searching on a user name for example, you can pull the followers field from the requester's ES document and query on those ids.
Now this approach works for small data sets, say less than 10,000 followers for example. But if you are a celebrity for example and have millions of followers this approach wont work, and I'm not talking about the maximum terms limit per query in elasticsearch.
Lets assume we did not have a max terms limit per query, if we want to search through 1M user records for example, we have to fetch the field from ES, then build a query with those terms and query again over 1M terms and pray ES does not crash or run out of memory in the process.
Not to mention that the approach taken in the example requires frequently updating the ES document for every follow and unfollow request and in my 5+ years experience in ES I would rather stay away from scripted updates at all costs.
So my question is, is there a known design pattern around this issue? what options to consider since there is no one-to-many relationship in ES.