# Getting list of terms ranked by their document frequency?

**URL:** <https://discuss.elastic.co/t/getting-list-of-terms-ranked-by-their-document-frequency/82346>\
**Category:** Elasticsearch\
**Created:** [April 13, 2017, 7:21pm UTC](https://discuss.elastic.co/t/getting-list-of-terms-ranked-by-their-document-frequency/82346 "2017-04-13T19:21:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![olalonde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olalonde/32/17118_2.png) [@olalonde](https://discuss.elastic.co/u/olalonde)\
**Post date:** [April 13, 2017, 7:21pm UTC](https://discuss.elastic.co/t/getting-list-of-terms-ranked-by-their-document-frequency/82346/1 "2017-04-13T19:21:25Z")

</div>

I'm using Elasticsearch to index images using the [bag of visual words](https://en.wikipedia.org/wiki/Bag-of-words_model_in_computer_vision) approach (the words are stored in a "visual\_words" field in image documents). Unfortunately, it seems like the inverted index isn't helping speed up my searches much. In my test, I have indexed 25000 images and a typical query returns \>24000 results. I'm guessing this is because a few of my dictionary words are present in almost all documents. Is there anything else that would cause Elasticsearch to return \>24000 results out of 25000? Also, is there a way for me to get a list of terms ordered by their document frequency so I see if my hypothesis is right and if this is the case for only a few words or lots of them.

Thanks!

---

<div class="post-metadata">

**Author:** ![olalonde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olalonde/32/17118_2.png) [@olalonde](https://discuss.elastic.co/u/olalonde)\
**Post date:** [April 14, 2017, 3:24am UTC](https://discuss.elastic.co/t/getting-list-of-terms-ranked-by-their-document-frequency/82346/2 "2017-04-14T03:24:19Z")

</div>

So I did some math and the issue really is the size of my dictionary:

 ![](https://us1.discourse-cdn.com/elastic/original/3X/6/8/6802c839e6cf8fc5c339fb67fb813cabd5501e9b.png)

Here's my math (in python): [https://gist.github.com/0c8589a67fcaaa32e4ccdbde1bfd6d3d](https://gist.github.com/0c8589a67fcaaa32e4ccdbde1bfd6d3d)

It might be wrong but seems to match the result I'm getting experimentally.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 12, 2017, 3:34am UTC](https://discuss.elastic.co/t/getting-list-of-terms-ranked-by-their-document-frequency/82346/3 "2017-05-12T03:34:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.