# Building a custom tokenizer: "Could not find suitable constructor"

**URL:** <https://discuss.elastic.co/t/building-a-custom-tokenizer-could-not-find-suitable-constructor/101138>\
**Category:** Elasticsearch\
**Created:** [September 20, 2017, 9:11am UTC](https://discuss.elastic.co/t/building-a-custom-tokenizer-could-not-find-suitable-constructor/101138 "2017-09-20T09:11:45Z")\
**Posts on this page:** 1\
**Showing post:** 14

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [September 22, 2017, 7:55pm UTC](https://discuss.elastic.co/t/building-a-custom-tokenizer-could-not-find-suitable-constructor/101138/14 "2017-09-22T19:55:23Z")

</div>

Yup. It's calling my tokenizer. But now it's revealed that my tokenizer is in fact crap!

```auto
Caused by: java.lang.IndexOutOfBoundsException
	at org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl.append(CharTermAttributeImpl.java:131)
	at com.cameraforensics.elasticsearch.plugins.UrlTokenizer.incrementToken(UrlTokenizer.java:30)

```

Probably because - as there are no docs - I'm doing it wrong.

```auto
    @Override
    public boolean incrementToken() throws IOException {
        if (position >= tokens.size()) {
            return false;
        } else {
            termAtt.setEmpty().append(tokens.get(position), position, position);
            position++;
            return true;
        }
    }

```

`tokens` is a list of all permutations of index segmentation (as per this: [Performance of doc\_values field vs analysed field](https://discuss.elastic.co/t/performance-of-doc-values-field-vs-analysed-field/99445/2))

I'm not really sure what the two int values should be on `CharTermAttribute#append`, so I'm guessing - incorrectly.

Anyway, thanks for all of your help. I'll keep hacking!

---

_[View the full topic](https://discuss.elastic.co/t/building-a-custom-tokenizer-could-not-find-suitable-constructor/101138)._