How to query on {brand} and/or {category string} efficiently?

I'm looking for assistance in searching our product index for terms such as:
Men's Shoes
Nike Men's Shoes
Men's Nike Shoes

For this specific query, the following fields are applicable:
brand_exact (non-tokenized, lowercase, asciifolding)
brand_t (tokenized with whitespace tokenizer, lowercase, asciifolding)
brand_ngram (edge n-gram 3 to 8 tokenizer, lowercase, asciifolding)
categories_t (english analyzer)

An example document might look like:
brand_exact: nike
brand_t: nike
brand_ngram, [nik, nike]
categories_t [men's shoes]

I've tried various combinations of matching using the bool filter and am able to produce relevant results; but I land up with too many irrelevant results at the end and am looking to improve this query or cutoff results.
A search on "Men's Shoes" brings up all men's shoes first, and then all other men's products because of the categories_t token match on 'men'.

Does anyone have any suggestions for how I could improve this?