Read this and specifically the "Also be patient" part.
Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.
A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.
Could you also provide the current output and the expected output?
I don't think you can order terms bucket by score as score does not make sense in a Terms Agg IMHO.