# Raw tf-idf **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2017, 8:54pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170 "2017-07-06T20:54:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![pierce](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@pierce](https://discuss.elastic.co/u/pierce)\ **Post date:** [July 6, 2017, 8:54pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/1 "2017-07-06T20:54:33Z")
I'm looking for a really scalable method to do scoring by tf-idf on a large dataset, so elastic search naturally came to mind. Crucially, I need raw tf-idf, versus the Practical Scoring Function that Lucene uses under the hood. Is there any way to get elastic search to return a raw tf-idf score without the additional fluff? I've tested each of the built-in implementations and none work as well as just tf-idf. So far I've investigated custom scoring functions, but this doesn't seem to be the right tool for the task. Also, I'm using a hosted elaticsearch instance, so I don't have access to the internals of the Java code. Only the REST API. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 9:28pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/2 "2017-07-06T21:28:15Z")
Elasticsearch uses BM25 as of 5.0. > [@pierce](#): > > So far I've investigated custom scoring functions, but this doesn't seem to be the right tool for the task. It is likely to be though, what put you off that? ---
**Author:** ![pierce](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@pierce](https://discuss.elastic.co/u/pierce)\ **Post date:** [July 6, 2017, 10:06pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/3 "2017-07-06T22:06:50Z")
All the examples seemed to be tied to the individual fields and looked like there wasn't any ability to extract the overall document frequency of each term. Would love to find a way to do this though. ---
**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\ **Post date:** [July 6, 2017, 10:32pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/4 "2017-07-06T22:32:19Z")
See the new advanced scripting docs which describe how to create a script engine, which then has access to lucene internals (you could get raw tf and idf). [https://www.elastic.co/guide/en/elasticsearch/reference/5.x/modules-scripting-engine.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.x/modules-scripting-engine.html) ---
**Author:** ![pierce](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@pierce](https://discuss.elastic.co/u/pierce)\ **Post date:** [July 6, 2017, 10:35pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/5 "2017-07-06T22:35:18Z")
Hi Ryan - is there a way to facilitate similar scripting only via REST? I have to use a remote elastic search client so don't have the ability to add a plugin. ---
**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\ **Post date:** [July 6, 2017, 10:50pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/6 "2017-07-06T22:50:45Z")
There is no other way to gain access to lucene internals. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 3, 2017, 10:50pm UTC](https://discuss.elastic.co/t/raw-tf-idf/92170/7 "2017-08-03T22:50:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.