You will probably find this talk by @Mark_Harwood interesting. Here he presents a way of answering these exact questions by indexing your documents in an "entity-centric" (in your case it would be user-centric) way
https://www.elastic.co/elasticon/2015/sf/building-entity-centric-indexes