# More accurate date based scoring

**URL:** <https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095>\
**Category:** Elasticsearch\
**Created:** [October 6, 2014, 5:17pm UTC](https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095 "2014-10-06T17:17:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\
**Post date:** [October 6, 2014, 5:17pm UTC](https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095/1 "2014-10-06T17:17:25Z")

</div>

PUT /test/test/1  
{  
"date":"2013-04-01T00:00:00Z"  
}

PUT /test/test/2  
{  
"date":"2013-04-01T00:00:01Z"  
}

PUT /test/test/3  
{  
"date":"2013-04-01T00:00:03Z"  
}

PUT /test/test/4  
{  
"date":"2013-04-01T00:01:03Z"  
}

Given these documents, I'm trying to come up with a query that scores them  
such that they come out in their natural sort order using function\_score.  
My problem is that ids 1 to 3 always come out with the exact same score.

GET /test/test/\_search  
{  
"query": {  
"function\_score": {  
"score\_mode": "max",  
"functions": [  
{  
"exp": {  
"date": {  
"origin": "2014-10-01T00:00:00Z",  
"scale": "1000d"  
}  
}  
}  
]  
}  
}  
}

This query is a good example.

I tried prototyping a script query which seems to reveal the real issue:  
the dates have an accuracy of 1 minute.

GET /test/\_search  
{  
"fields": ["date"],  
"query": {  
"function\_score": {  
"query": {  
"match\_all": {}  
},  
"score\_mode": "max",  
"functions": [  
{  
"script\_score": {  
"lang": "expression",  
"script": "doc['date'].value"  
}  
}  
]  
}  
}  
}

This query returns the actual field value as the score. For the first three  
documents I get the score: 1364774350000, the fourth document is scored  
with 1364774490000. It looks very much like elasticsearch is rounding the  
timestamp internally.

Is there a way to get second level accuracy (or even better) here? I know I  
can use sorting here but that would effectively get rid of any meaningful  
ranking for the rest of my query. But minute accuracy is just not going to  
be good enough either.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [October 6, 2014, 6:42pm UTC](https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095/2 "2014-10-06T18:42:10Z")

</div>

This is not Elasticsearch.

Timestamps are represented in milliseconds as 64-bit longs. Scores are  
represented as 64-bit doubles. Per Java language specification, it is not  
possible to express a 64 bit long in a double because IEEE 754 only allows  
52 bits.

> **[Double-precision floating-point format](https://en.wikipedia.org/wiki/Double-precision_floating-point_format)**
>
> Double-precision floating-point format (sometimes called FP64 or float64) is a floating-point number format, usually occupying 64 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point.
> Double precision may be chosen when the range or precision of single precision would be insufficient.
> In the IEEE 754-2008 standard, the 64-bit base-2 format is officially referred to as binary64; it was called double in IEEE 754-1985. IEEE 754 specifies ad...

So all longs with a finer resolution than 52 bit (that is second  
resolution) can not be used as an injective function for scoring.

You have the following options:

- use a coarser resolution than second

- organize timestamps into two or more fields for script based calculations  
(e.g. days since 01-Jan-1970, seconds of day)

- use only a subrange of all possible timestamps between Thu Jan 01  
01:00:00 CET 1970 and Sun Aug 17 08:12:55 CET 292278994 that can map into  
52 bit wide longs so they can be represented by doubles

Jörg

On Mon, Oct 6, 2014 at 7:17 PM, Jilles van Gurp [jillesvangurp@gmail.com](mailto:jillesvangurp@gmail.com)  
wrote:

> PUT /test/test/1  
> {  
> "date":"2013-04-01T00:00:00Z"  
> }
> 
> PUT /test/test/2  
> {  
> "date":"2013-04-01T00:00:01Z"  
> }
> 
> PUT /test/test/3  
> {  
> "date":"2013-04-01T00:00:03Z"  
> }
> 
> PUT /test/test/4  
> {  
> "date":"2013-04-01T00:01:03Z"  
> }
> 
> Given these documents, I'm trying to come up with a query that scores them  
> such that they come out in their natural sort order using function\_score.  
> My problem is that ids 1 to 3 always come out with the exact same score.
> 
> GET /test/test/\_search  
> {  
> "query": {  
> "function\_score": {  
> "score\_mode": "max",  
> "functions": [  
> {  
> "exp": {  
> "date": {  
> "origin": "2014-10-01T00:00:00Z",  
> "scale": "1000d"  
> }  
> }  
> }  
> ]  
> }  
> }  
> }
> 
> This query is a good example.
> 
> I tried prototyping a script query which seems to reveal the real issue:  
> the dates have an accuracy of 1 minute.
> 
> GET /test/\_search  
> {  
> "fields": ["date"],  
> "query": {  
> "function\_score": {  
> "query": {  
> "match\_all": {}  
> },  
> "score\_mode": "max",  
> "functions": [  
> {  
> "script\_score": {  
> "lang": "expression",  
> "script": "doc['date'].value"  
> }  
> }  
> ]  
> }  
> }  
> }
> 
> This query returns the actual field value as the score. For the first  
> three documents I get the score: 1364774350000, the fourth document is  
> scored with 1364774490000. It looks very much like elasticsearch is  
> rounding the timestamp internally.
> 
> Is there a way to get second level accuracy (or even better) here? I know  
> I can use sorting here but that would effectively get rid of any meaningful  
> ranking for the rest of my query. But minute accuracy is just not going to  
> be good enough either.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE6eL-GS%2BdYh6SuoJ1Qy6sdPWhqGttGUEEZrr0Z3SgSmg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE6eL-GS%2BdYh6SuoJ1Qy6sdPWhqGttGUEEZrr0Z3SgSmg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\
**Post date:** [October 7, 2014, 8:09am UTC](https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095/3 "2014-10-07T08:09:27Z")

</div>

Thanks for the clarification. I'll have to experiment a bit with this.

How would the multi field solution work on the ranking side? You'd still  
have the precision problem when combining the scores, right?

Jilles

On Monday, October 6, 2014 8:42:17 PM UTC+2, Jörg Prante wrote:

> This is not Elasticsearch.
> 
> Timestamps are represented in milliseconds as 64-bit longs. Scores are  
> represented as 64-bit doubles. Per Java language specification, it is not  
> possible to express a 64 bit long in a double because IEEE 754 only allows  
> 52 bits.
> 
> [Double-precision floating-point format - Wikipedia](http://en.wikipedia.org/wiki/Double-precision_floating-point_format)
> 
> So all longs with a finer resolution than 52 bit (that is second  
> resolution) can not be used as an injective function for scoring.
> 
> You have the following options:
> 
> - use a coarser resolution than second
> 
> - organize timestamps into two or more fields for script based  
> calculations (e.g. days since 01-Jan-1970, seconds of day)
> 
> - use only a subrange of all possible timestamps between Thu Jan 01  
> 01:00:00 CET 1970 and Sun Aug 17 08:12:55 CET 292278994 that can map into  
> 52 bit wide longs so they can be represented by doubles
> 
> Jörg
> 
> On Mon, Oct 6, 2014 at 7:17 PM, Jilles van Gurp \<[jilles...@gmail.com](mailto:jilles...@gmail.com)  
> \<javascript:\>\> wrote:
> 
> > PUT /test/test/1  
> > {  
> > "date":"2013-04-01T00:00:00Z"  
> > }
> > 
> > PUT /test/test/2  
> > {  
> > "date":"2013-04-01T00:00:01Z"  
> > }
> > 
> > PUT /test/test/3  
> > {  
> > "date":"2013-04-01T00:00:03Z"  
> > }
> > 
> > PUT /test/test/4  
> > {  
> > "date":"2013-04-01T00:01:03Z"  
> > }
> > 
> > Given these documents, I'm trying to come up with a query that scores  
> > them such that they come out in their natural sort order using  
> > function\_score. My problem is that ids 1 to 3 always come out with the  
> > exact same score.
> > 
> > GET /test/test/\_search  
> > {  
> > "query": {  
> > "function\_score": {  
> > "score\_mode": "max",  
> > "functions": [  
> > {  
> > "exp": {  
> > "date": {  
> > "origin": "2014-10-01T00:00:00Z",  
> > "scale": "1000d"  
> > }  
> > }  
> > }  
> > ]  
> > }  
> > }  
> > }
> > 
> > This query is a good example.
> > 
> > I tried prototyping a script query which seems to reveal the real issue:  
> > the dates have an accuracy of 1 minute.
> > 
> > GET /test/\_search  
> > {  
> > "fields": ["date"],  
> > "query": {  
> > "function\_score": {  
> > "query": {  
> > "match\_all": {}  
> > },  
> > "score\_mode": "max",  
> > "functions": [  
> > {  
> > "script\_score": {  
> > "lang": "expression",  
> > "script": "doc['date'].value"  
> > }  
> > }  
> > ]  
> > }  
> > }  
> > }
> > 
> > This query returns the actual field value as the score. For the first  
> > three documents I get the score: 1364774350000, the fourth document is  
> > scored with 1364774490000. It looks very much like elasticsearch is  
> > rounding the timestamp internally.
> > 
> > Is there a way to get second level accuracy (or even better) here? I know  
> > I can use sorting here but that would effectively get rid of any meaningful  
> > ranking for the rest of my query. But minute accuracy is just not going to  
> > be good enough either.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5c8b7caa-06d5-4992-a466-b9cc4a8f397b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/39232802-4430-415f-9217-39128493eb70%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/39232802-4430-415f-9217-39128493eb70%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\
**Post date:** [October 7, 2014, 8:16am UTC](https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095/4 "2014-10-07T08:16:38Z")

</div>

I found another workable solution.

"sort": [  
{  
"\_score": {  
"order": "desc"  
}  
},  
{  
"date": {  
"order": "desc"  
}  
}  
]

This sorts first by score and then by date. So this has the effect of  
ranking by score and then ranking those items with the same score by date.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/150e9e85-3124-42d8-a4e9-a206c5280bfc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/150e9e85-3124-42d8-a4e9-a206c5280bfc%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/more-accurate-date-based-scoring/20095/5 "2017-07-06T00:57:57Z")

</div>