# Creating a tf-idf for unigrams in a corpus using elasticsearch?

**URL:** <https://discuss.elastic.co/t/creating-a-tf-idf-for-unigrams-in-a-corpus-using-elasticsearch/10334>\
**Category:** Elasticsearch\
**Created:** [January 14, 2013, 5:41pm UTC](https://discuss.elastic.co/t/creating-a-tf-idf-for-unigrams-in-a-corpus-using-elasticsearch/10334 "2013-01-14T17:41:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\
**Post date:** [January 14, 2013, 5:41pm UTC](https://discuss.elastic.co/t/creating-a-tf-idf-for-unigrams-in-a-corpus-using-elasticsearch/10334/1 "2013-01-14T17:41:51Z")

</div>

Hi Everyone,  
I'm interested in creating a tf-idf matrix for unigrams in a corpus of  
documents I have stored in elastic search.

I have searched the list, and from the results, I think that this perhaps  
is not possible out of the box using the elasticsearch API, but wanted to  
confirm before I start coding up a solution. Is there any way to calculate  
tf-idf for every term in a corpus using the API?

I hope the question is clear, I'm a bit new to this space, and learning as  
I go.

thanks,

--paul

--

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [January 15, 2013, 3:34am UTC](https://discuss.elastic.co/t/creating-a-tf-idf-for-unigrams-in-a-corpus-using-elasticsearch/10334/2 "2013-01-15T03:34:00Z")

</div>

There is a plugin that will return the term frequences:

> **[GitHub - jprante/elasticsearch-index-termlist: Elasticsearch Index Termlist](https://github.com/jprante/elasticsearch-index-termlist)**
>
> Elasticsearch Index Termlist. Contribute to jprante/elasticsearch-index-termlist development by creating an account on GitHub.

You can see how the plugin uses standard Lucene code to calculate the term  
frequencies at:  
[https://github.com/jprante/elasticsearch-index-termlist/blob/master/src/main/java/org/elasticsearch/action/termlist/TransportTermlistAction.java#L126](https://github.com/jprante/elasticsearch-index-termlist/blob/master/src/main/java/org/elasticsearch/action/termlist/TransportTermlistAction.java#L126)

IDF would be trickier. Not sure if anything is exposed to calculate the  
IDF. The difficulty in Elasticsearch is that an index is distributed, so no  
Lucene index has all of the terms.

Cheers,

Ivan

On Mon, Jan 14, 2013 at 9:41 AM, Paul Sanwald [paul@redowlanalytics.com](mailto:paul@redowlanalytics.com)wrote:

> Hi Everyone,  
> I'm interested in creating a tf-idf matrix for unigrams in a corpus of  
> documents I have stored in Elasticsearch.
> 
> I have searched the list, and from the results, I think that this perhaps  
> is not possible out of the box using the elasticsearch API, but wanted to  
> confirm before I start coding up a solution. Is there any way to calculate  
> tf-idf for every term in a corpus using the API?
> 
> I hope the question is clear, I'm a bit new to this space, and learning as  
> I go.
> 
> thanks,
> 
> --paul
> 
> --

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:56am UTC](https://discuss.elastic.co/t/creating-a-tf-idf-for-unigrams-in-a-corpus-using-elasticsearch/10334/3 "2017-07-06T02:56:29Z")

</div>