# Elasticsearch token\_vector analysis over an entire field **URL:** \ **Category:** Elasticsearch\ **Created:** [September 19, 2017, 2:39pm UTC](https://discuss.elastic.co/t/elasticsearch-token-vector-analysis-over-an-entire-field/101036 "2017-09-19T14:39:14Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![MPurcell](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@MPurcell](https://discuss.elastic.co/u/MPurcell)\ **Post date:** [September 19, 2017, 2:39pm UTC](https://discuss.elastic.co/t/elasticsearch-token-vector-analysis-over-an-entire-field/101036/1 "2017-09-19T14:39:14Z")
Hello I'm writing this question in the hopes for some clarity regarding token\_vector analysis. I am looking to get a count of unique tokens in a text/keyword field. So for example in the token\_vector query below: GET /twitter/tweet/1/\_termvectors { "fields" : ["text"], "offsets" : true, "payloads" : true, "positions" : true, "term\_statistics" : true, "field\_statistics" : true } The response I receive is a count of the tokens within the document. Instead what I would like is a count of the different tokens over an entire field in the index -- not a single document. Instead of this response: { "\_id": "1", "\_index": "twitter", "\_type": "tweet", "\_version": 1, "found": true, "took": 6, "term\_vectors": { "text": { "field\_statistics": { "doc\_count": 2, "sum\_doc\_freq": 6, "sum\_ttf": 8 }, "terms": { "test": { "doc\_freq": 2, "term\_freq": 3, "tokens": [ { "end\_offset": 12, "payload": "d29yZA==", "position": 1, "start\_offset": 8 }, { "end\_offset": 17, "payload": "d29yZA==", "position": 2, "start\_offset": 13 }, { "end\_offset": 22, "payload": "d29yZA==", "position": 3, "start\_offset": 18 } ], "ttf": 4 }, "twitter": { "doc\_freq": 2, "term\_freq": 1, "tokens": [ { "end\_offset": 7, "payload": "d29yZA==", "position": 0, "start\_offset": 0 } ], "ttf": 2 } } } } } I would like just term counts over an entire index. In trying to figure this out I have loaded a field as both text and keyword (it's a list of addresses). What I'm looking for is a count of all unique terms within this Address field. My hope is the response from ES would be something like this: { "index" : addresses, "type" : by\_your\_house" "terms" : { "ROAD" {} "STREET" {} "LANE" {} }} I have tried using kibana for this task but it will not split up the address terms correctly. It will instead show aggregations of entire Street names that are common. So instead of above I, what I see in Kibana is: { "1007 Mountain Drive" : 99 "20 Ingram Street" : 55 " 1938 Sullivan Lane" : 11 } Thanks for any/all help with this. - Matt ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 20, 2017, 12:26am UTC](https://discuss.elastic.co/t/elasticsearch-token-vector-analysis-over-an-entire-field/101036/2 "2017-09-20T00:26:32Z")
The only way to get the term vectors over the entire index is to use an aggregation over the analyzed field. If you are on a "modern" version of Elasticsearch, you would need to enabled fielddata on the field for aggregations to work: [https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html) Field data is expensive, so it is not something to use lightly. Do you just need the number of terms, without their values, you can use the cardinarlity or value count aggregations: [https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-aggregations-metrics-cardinality-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-aggregations-metrics-cardinality-aggregation.html) [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-valuecount-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-valuecount-aggregation.html) ---
**Author:** ![MPurcell](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@MPurcell](https://discuss.elastic.co/u/MPurcell)\ **Post date:** [September 20, 2017, 3:35pm UTC](https://discuss.elastic.co/t/elasticsearch-token-vector-analysis-over-an-entire-field/101036/3 "2017-09-20T15:35:03Z")
Great so this brings me about halfway there! I ran a simple aggs query after setting fielddata : True on the City field. The query was as follows: GET /my\_index/\_search { "size" : 0, "aggs" : { "states" : { "terms" : { "field" : "City" } } } } and the response: }, "aggregations": { "states": { "doc\_count\_error\_upper\_bound": 117547, "sum\_other\_doc\_count": 28080179, "buckets": [ { "key": "san", "doc\_count": 427575 }, { "key": "new", "doc\_count": 414354 }, { "key": "chicago", "doc\_count": 265567 }, { "key": "lake", "doc\_count": 245682 }, { "key": "beach", "doc\_count": 219230 }, { "key": "park", "doc\_count": 215134 }, { "key": "york", "doc\_count": 208854 }, { "key": "west", "doc\_count": 188746 }, { "key": "portland", "doc\_count": 188657 }, { "key": "fort", "doc\_count": 183512 } ] } } } What I would like to do is be able to get all the results and not just the first 10. Visualizing this in Kibana would probably work, but an aggs query might work just as well for finding the terms and their counts that I'm looking for. Is there a best way to go about this? Thanks for all your help. ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [September 20, 2017, 4:38pm UTC](https://discuss.elastic.co/t/elasticsearch-token-vector-analysis-over-an-entire-field/101036/4 "2017-09-20T16:38:57Z")
Bucket aggregations have a size parameters that will control the number of results returned. If not, the default is 10. Depending on the size of your corpus, you might have thousands of unique tokens. Since your first ten have a minimum of 183K, the number will probably be high. One thing you might have noticed is that the terms are the analyzed tokens, which might be a limiting factor if you need the original text. Cheers, Ivan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 18, 2017, 4:38pm UTC](https://discuss.elastic.co/t/elasticsearch-token-vector-analysis-over-an-entire-field/101036/5 "2017-10-18T16:38:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.