# How to use Limit Token Filter? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 21, 2017, 4:17pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913 "2017-02-21T16:17:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mathmatomancer](https://avatars.discourse-cdn.com/v4/letter/m/ee59a6/32.png) [@mathmatomancer](https://discuss.elastic.co/u/mathmatomancer)\ **Post date:** [February 21, 2017, 4:17pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/1 "2017-02-21T16:17:09Z")
Hi, I'm trying to use the limit token filter to cap the amount of data stored in elasticsearch from a potentially very large text string in my main database. I have in my settings this filter and analyzer specified: ``` analysis": { "filter": { "max_size_tokens": { "type": "limit", "max_token_count": "50" } }, "analyzer": { "large_text_blobs": { "filter": [ "lowercase", "unique", "max_size_tokens", ], "type": "custom", "tokenizer": "standard" } } ``` but analyzing a long string is returning more than the 50 tokens I specified. (Example string is just the numbers 1 - 200, with comma and space between; it returns all 200 tokens.) BTW this is on elasticsearch 5.2.1, running locally, and the consume\_all\_tokens option does not appear to have any effect on this. ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [February 22, 2017, 6:05pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/2 "2017-02-22T18:05:26Z")
The limit token filter will only affect the indexed value, not what is returned in results via the \_source. If you want to limit the field in general you should look into using an ingest pipeline which would be applied before indexing. ---
**Author:** ![mathmatomancer](https://avatars.discourse-cdn.com/v4/letter/m/ee59a6/32.png) [@mathmatomancer](https://discuss.elastic.co/u/mathmatomancer)\ **Post date:** [February 22, 2017, 6:35pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/3 "2017-02-22T18:35:18Z")
Yes, I'm not looking to change the source value. The extra tokens I'm seeing are from looking at the results of \_analyze using the specified filter: ``` curl "http://localhost:9200/customers/_analyze?analyzer=large_text_blobs&index=customers&text=1%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%2C+11%2C+12%2C+13%2C+14%2C+15%2C+16%2C+17%2C+18%2C+19%2C+20%2C+21%2C+22%2C+23%2C+24%2C+25%2C+26%2C+27%2C+28%2C+29%2C+30%2C+31%2C+32%2C+33%2C+34%2C+35%2C+36%2C+37%2C+38%2C+39%2C+40%2C+41%2C+42%2C+43%2C+44%2C+45%2C+46%2C+47%2C+48%2C+49%2C+50%2C+51%2C+52%2C+53%2C+54%2C+55%2C+56%2C+57%2C+58%2C+59%2C+60%2C+61%2C+62%2C+63%2C+64%2C+65%2C+66%2C+67%2C+68%2C+69%2C+70%2C+71%2C+72%2C+73%2C+74%2C+75%2C+76%2C+77%2C+78%2C+79%2C+80%2C+81%2C+82%2C+83%2C+84%2C+85%2C+86%2C+87%2C+88%2C+89%2C+90%2C+91%2C+92%2C+93%2C+94%2C+95%2C+96%2C+97%2C+98%2C+99%2C+100%2C+101%2C+102%2C+103%2C+104%2C+105%2C+106%2C+107%2C+108%2C+109%2C+110%2C+111%2C+112%2C+113%2C+114%2C+115%2C+116%2C+117%2C+118%2C+119%2C+120%2C+121%2C+122%2C+123%2C+124%2C+125%2C+126%2C+127%2C+128%2C+129%2C+130%2C+131%2C+132%2C+133%2C+134%2C+135%2C+136%2C+137%2C+138%2C+139%2C+140%2C+141%2C+142%2C+143%2C+144%2C+145%2C+146%2C+147%2C+148%2C+149%2C+150%2C+151%2C+152%2C+153%2C+154%2C+155%2C+156%2C+157%2C+158%2C+159%2C+160%2C+161%2C+162%2C+163%2C+164%2C+165%2C+166%2C+167%2C+168%2C+169%2C+170%2C+171%2C+172%2C+173%2C+174%2C+175%2C+176%2C+177%2C+178%2C+179%2C+180%2C+181%2C+182%2C+183%2C+184%2C+185%2C+186%2C+187%2C+188%2C+189%2C+190%2C+191%2C+192%2C+193%2C+194%2C+195%2C+196%2C+197%2C+198%2C+199%2C+200" ``` This returns the full list of tokens, where I'd expect only the first 50 (or really any 50, it doesn't matter for this). ``` {"token":"1","start_offset":0,"end_offset":1,"type":"","position":0}, ... {"token":"199","start_offset":882,"end_offset":885,"type":"","position":19998}, {"token":"200","start_offset":887,"end_offset":890,"type":"","position":20099}]} ``` ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [February 22, 2017, 7:12pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/4 "2017-02-22T19:12:02Z")
I only have a 5.2.0 instance to test with right this second but using analyze api it works: ```auto curl -XPOST 'http://localhost:9200/_analyze' -d '{ "tokenizer": "standard", "filter": ["lowercase", "unique", {"type": "limit", "max_token_count": 5}], "text": "1, 2, 3, 4, 5, 6, 7, 8, 9, 10" }' ``` Gives me: ```auto { "tokens": [ { "token": "1", "start_offset": 0, "end_offset": 1, "type": "", "position": 0 }, { "token": "2", "start_offset": 3, "end_offset": 4, "type": "", "position": 1 }, { "token": "3", "start_offset": 6, "end_offset": 7, "type": "", "position": 2 }, { "token": "4", "start_offset": 9, "end_offset": 10, "type": "", "position": 3 }, { "token": "5", "start_offset": 12, "end_offset": 13, "type": "", "position": 4 } ] } ``` ---
**Author:** ![mathmatomancer](https://avatars.discourse-cdn.com/v4/letter/m/ee59a6/32.png) [@mathmatomancer](https://discuss.elastic.co/u/mathmatomancer)\ **Post date:** [February 22, 2017, 9:40pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/5 "2017-02-22T21:40:43Z")
Hmm, thanks for pointing me at that, your data looks like it's from Kibana... It looks like the curl generated by 'copy as curl' from kibana is formatted differently than my ruby gem's .analyze method does it, and for some reason that breaks it? From Kibana (returns the expected amount of data): ``` curl -XPOST "http://localhost:9200/help_articles/_analyze" -d' { "text": "1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100", "analyzer": "large_text_blobs" }' ``` If I use the query string, even in Kibana, it returns the full set of tokens instead of limiting it. ``` curl -XGET "http://localhost:9200/help_articles/_analyze?analyzer=large_text_blobs&index=help_articles&text=1%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%2C+11%2C+12%2C+13%2C+14%2C+15%2C+16%2C+17%2C+18%2C+19%2C+20%2C+21%2C+22%2C+23%2C+24%2C+25%2C+26%2C+27%2C+28%2C+29%2C+30%2C+31%2C+32%2C+33%2C+34%2C+35%2C+36%2C+37%2C+38%2C+39%2C+40..." ``` ---
**Author:** ![mathmatomancer](https://avatars.discourse-cdn.com/v4/letter/m/ee59a6/32.png) [@mathmatomancer](https://discuss.elastic.co/u/mathmatomancer)\ **Post date:** [February 23, 2017, 4:27pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/6 "2017-02-23T16:27:44Z")
Okay, in case anyone else runs into similar problems, the query string version of this is deprecated, and while it's surprising that it doesn't work, it's not going to be supported in the future anyway. The ruby gem does actually do either, depending on how you make the call, and the details of that are a bit hidden. Incorrect (generates the url-params version): ``` @client.indices.analyze(text: 'long string', analyzer: :large_text_blobs, index: 'help_articles') ``` Correct : ``` @client.indices.analyze(body: {text: 'long string', analyzer: :large_text_blobs}, index: 'help_articles') ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 23, 2017, 4:27pm UTC](https://discuss.elastic.co/t/how-to-use-limit-token-filter/75913/7 "2017-03-23T16:27:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.