What would cause refresh=wait_for to regularly take 2-5 seconds?

Are you using nested documents or parent-child relationships in your indices? If so, this thread may be applicable.