I suppose you are using the nested query. You may want to take a look at the score_mode - I'm however not sure if that is exactly what you are searching for.
Sharing your mapping, index creation and query would probably help!.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.