This might be useful references:
For me, it was almost always fielddata. Look for evictions in that and filterdata. _node/stats and/or marvel or other monitor is your friend for troubleshooting this.
If you are not using doc_values, it can help (at some indexing cost).