I suppose the only way to use context and timefield filtering is to implement the query using only aggregation.
If you used query clause to filter documents, you can use Filter aggregation instead of query clause.
If you use "hits" of the query for formatter in vega-lite, it seems difficult but using filter aggregation with scripted metric aggregation "trick" to retrieve all documents in a bucket could be a possible choice though I have not tried.
I see. Please check the Request and Response in Inspector. Are there buckets as you expected?