Slow Custom score query

Hi All,
I'm using ElasticSearch to search through a dataset of around 1.2M records, with 25 fields each.
When I do a simple query, by 5/6 terms, almost instantly.
My problems start when I try to add a custom score to the same query, with a simple script like: "(1/(time - doc['time_field'].value)) * _score" , the request now takes around 5-10 seconds to complete.
Is it normal for the custom score query to take that long?
I'm running ElasticSearch on a dual core system, with 8gb of memory.

Thanks in advance,
Best Regards
Ariel