This sounds related to the terms returned per shard. Might be worth reading through this discuss issue which includes some helpful links on how to account for and resolve issues related to smaller queries and their interplay with shard size:
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.