My bad, didn't mean to offend anyone, I had all genders in mind to be honest. I would edit the post to include gals but I can't anymore. 
What exactly do you mean "handle this in your client logic"?
If you mean to filter out most data on the client side instead of Elasticsearch then no, there will be too much data eventually and it will make the user experience very bad.
Besides, if this would be an option, then why use Elasticsearch at all.