How to custom score (score based on a date field)?


(John Chang) #1

I'd like to have documents come back sorted by a date saved in my doc. So if I ask for the top 100 hits, I want the 100 most recent according to a doc field, not according to the time time of indexing. I did find the discussion titled "Recency" in this forum which suggested it might be coming (in April 2010 kimchy wrote he intended to add it).

Is it in 0.7.1 by chance? If not, do you know when it might come? Any suggestions in the meantime for how to get this to work?

Thanks for your time,
John


(Shay Banon) #2

I am not really sure I understand, why can't you use sorting?

On Tue, May 25, 2010 at 10:29 PM, John Chang jchangkihtest2@gmail.comwrote:

I'd like to have documents come back sorted by a date saved in my doc. So
if
I ask for the top 100 hits, I want the 100 most recent according to a doc
field, not according to the time time of indexing. I did find the
discussion titled "Recency" in this forum which suggested it might be
coming
(in April 2010 kimchy wrote he intended to add it).

Is it in 0.7.1 by chance? If not, do you know when it might come? Any
suggestions in the meantime for how to get this to work?

Thanks for your time,
John

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/How-to-custom-score-score-based-on-a-date-field-tp842971p842971.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Clinton Gormley) #3

On Wed, 2010-05-26 at 09:28 +0300, Shay Banon wrote:

I am not really sure I understand, why can't you use sorting?

I think this relates to my previous thread about using 'recency' as part
of the relevance weighting.

So a search for Joe Bloggs should give more weight to a document created
recently, than one a year old.

clint

On Tue, May 25, 2010 at 10:29 PM, John Chang
jchangkihtest2@gmail.com wrote:

    I'd like to have documents come back sorted by a date saved in
    my doc.  So if
    I ask for the top 100 hits, I want the 100 most recent
    according to a doc
    field, not according to the time time of indexing.  I did find
    the
    discussion titled "Recency" in this forum which suggested it
    might be coming
    (in April 2010 kimchy wrote he intended to add it).
    
    Is it in 0.7.1 by chance?  If not, do you know when it might
    come?  Any
    suggestions in the meantime for how to get this to work?
    
    Thanks for your time,
    John
    --
    View this message in context:
    http://elasticsearch-users.115913.n3.nabble.com/How-to-custom-score-score-based-on-a-date-field-tp842971p842971.html
    Sent from the ElasticSearch Users mailing list archive at
    Nabble.com.

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.


(Shay Banon) #4

If it does, then this is not yet in elasticsearch. Was just wondering if the
question is more basic about sorting.

On Wed, May 26, 2010 at 1:43 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

On Wed, 2010-05-26 at 09:28 +0300, Shay Banon wrote:

I am not really sure I understand, why can't you use sorting?

I think this relates to my previous thread about using 'recency' as part
of the relevance weighting.

So a search for Joe Bloggs should give more weight to a document created
recently, than one a year old.

clint

On Tue, May 25, 2010 at 10:29 PM, John Chang
jchangkihtest2@gmail.com wrote:

    I'd like to have documents come back sorted by a date saved in
    my doc.  So if
    I ask for the top 100 hits, I want the 100 most recent
    according to a doc
    field, not according to the time time of indexing.  I did find
    the
    discussion titled "Recency" in this forum which suggested it
    might be coming
    (in April 2010 kimchy wrote he intended to add it).

    Is it in 0.7.1 by chance?  If not, do you know when it might
    come?  Any
    suggestions in the meantime for how to get this to work?

    Thanks for your time,
    John
    --
    View this message in context:

http://elasticsearch-users.115913.n3.nabble.com/How-to-custom-score-score-based-on-a-date-field-tp842971p842971.html

    Sent from the ElasticSearch Users mailing list archive at
    Nabble.com.

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.


(John Chang) #5

Sounds like it is not yet in Elastic Search. But, for the record, it is related to recency, but not exactly recency as discussed above. I want to sort based on a datetime field in the document, not on the datetime it was indexed. Say, sorting by birthdate when the order of indexing is not related to order of birth. So, the most recently indexed document is not necessarily the one with the most "recency" as concerns the query.


(John Chang) #6

Kimchy,

We have been testing out Elastic Search and are very impressed. However, we do need custom scoring based on a specific document date field; this is a must-have for V1 for us. We have a custom scoring solution we like with our straight-lucene (single box) setup and are now looking into how we could integrate that into Elastic Search. Ideally, we would like to make it a flexible solution that could be pulled into the master.

I didn't see any pulls into master for elasticsearch. Is this something you are open to?

Thanks for your time,

John Chang


(Shay Banon) #7

First, I am very much open for contributions. But, I am working as I type on
building the infra to support custom scoring, dynamic facets, range facets,
bucket facets, custom return fields, and map reduce (thats a mouthful :slight_smile: ).
It all revolves around the same infra.

I hope that within several days, you will see something.

-shay.banon

On Wed, Jun 2, 2010 at 9:05 PM, John Chang jchangkihtest2@gmail.com wrote:

Kimchy,

We have been testing out Elastic Search and are very impressed. However,
we
do need custom scoring based on a specific document date field; this is a
must-have for V1 for us. We have a custom scoring solution we like with
our
straight-lucene (single box) setup and are now looking into how we could
integrate that into Elastic Search. Ideally, we would like to make it a
flexible solution that could be pulled into the master.

I didn't see any pulls into master for elasticsearch. Is this something
you
are open to?

Thanks for your time,

John Chang

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/How-to-custom-score-score-based-on-a-date-field-tp842971p865496.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Shay Banon) #8

As a side note, is your custom implementation revolves around Lucene built
in function queries?

On Wed, Jun 2, 2010 at 11:35 PM, Shay Banon shay.banon@elasticsearch.comwrote:

First, I am very much open for contributions. But, I am working as I type
on building the infra to support custom scoring, dynamic facets, range
facets, bucket facets, custom return fields, and map reduce (thats a
mouthful :slight_smile: ). It all revolves around the same infra.

I hope that within several days, you will see something.

-shay.banon

On Wed, Jun 2, 2010 at 9:05 PM, John Chang jchangkihtest2@gmail.comwrote:

Kimchy,

We have been testing out Elastic Search and are very impressed. However,
we
do need custom scoring based on a specific document date field; this is a
must-have for V1 for us. We have a custom scoring solution we like with
our
straight-lucene (single box) setup and are now looking into how we could
integrate that into Elastic Search. Ideally, we would like to make it a
flexible solution that could be pulled into the master.

I didn't see any pulls into master for elasticsearch. Is this something
you
are open to?

Thanks for your time,

John Chang

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/How-to-custom-score-score-based-on-a-date-field-tp842971p865496.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(John Chang) #9

It is based on org.apache.lucene.search.function.FieldScoreQuery and and our subclasses of org.apache.lucene.search.function.CustomScoreQuery org.apache.lucene.search.function.CustomScoreProvider.

We are open to other solutions that work; the final solution doesn't have to be as described above, as long as we can score on a document field of our choosing and it works well.

Thanks -- I'll keep my eyes open for something new coming out soon.


(John Chang) #10

Oh, and I should add that although our current solution works acceptably, it is not perfect. We want to sort based on a date field in the document, which is stored as a long (java.util.Date.getTime()). As you probably know, Lucene CustomScoreProvider (and our subclasses) must return scores as a float, so we loose some precision in our scores. It is good enough for now, but it would be good to implement something better before long.


(John Chang) #11

Thanks. Please notify when it is in master, even if it is not yet in release, as we'd be happy to build it and try it out from there. We could at least help with testing, then.


(Shay Banon) #12

cool, I will notify you once its up.

On Thu, Jun 3, 2010 at 9:17 PM, John Chang jchangkihtest2@gmail.com wrote:

Thanks. Please notify when it is in master, even if it is not yet in
release, as we'd be happy to build it and try it out from there. We could
at least help with testing, then.

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/How-to-custom-score-score-based-on-a-date-field-tp842971p868331.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(system) #13