# Custom tokenizer doesn't work on reindex/index api, only \_analyze endpoint

**URL:** <https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715>\
**Category:** Elasticsearch\
**Created:** [September 25, 2017, 2:04pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715 "2017-09-25T14:04:51Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [September 25, 2017, 2:04pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/1 "2017-09-25T14:04:51Z")

</div>

I've written a custom tokenizer. It was hard ([Building a custom tokenizer: "Could not find suitable constructor"](https://discuss.elastic.co/t/building-a-custom-tokenizer-could-not-find-suitable-constructor/101138/8)), nevertheless I persevered.

I created an index with it configured:

```auto
      "analysis":{
            "analyzer":{
                "urlanalyzer":{ 
                    "type": "custom",
                    "tokenizer": "urltokenizer",
                    "filter": "urlstopwords"
                }
            },
            "filter": {
                "urlstopwords": {
                    "type": "stop",
                    "stopwords": ["http:", "https:"]
                }
            }
        }

```

When I run the `_analyze` tester against it, it all works, for example:

```auto
POST /myindex/_analyze
{
    "text": "https://www.domain.com/one/two/three/four.jpg",
    "tokenizer": "urltokenizer"
}

```

Response:

```auto
{
    "tokens": [
        {
            "token": "https://www.domain.com/one/two/three/four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 0
        },
        {
            "token": "www.domain.com/one",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 1
        },
        {
            "token": "www.domain.com/one/two",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 2
        },
        {
            "token": "three/four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 3
        },
        {
            "token": "www.domain.com/one/two/three",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 4
        },
        {
            "token": "one",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 5
        },
        {
            "token": "https://www.domain.com/one",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 6
        },
        {
            "token": "https://www.domain.com/one/two/three",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 7
        },
        {
            "token": "one/two",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 8
        },
        {
            "token": "two",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 9
        },
        {
            "token": "three",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 10
        },
        {
            "token": "https://www.domain.com/one/two",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 11
        },
        {
            "token": "www.domain.com/one/two/three/four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 12
        },
        {
            "token": "one/two/three/four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 13
        },
        {
            "token": "two/three/four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 14
        },
        {
            "token": "/one/two/three/four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 15
        },
        {
            "token": "two/three",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 16
        },
        {
            "token": "one/two/three",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 17
        },
        {
            "token": "four.jpg",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 18
        },
        {
            "token": "www.domain.com",
            "start_offset": 0,
            "end_offset": 0,
            "type": "word",
            "position": 19
        }
    ]
}

```

(_Should_ the offsets all be 0?)

If I check the term vectors using a test document, it comes back with these terms.

However, if I index a document into an index that uses the analyzer with the following mappings:

```auto
...
               "source": {
                    "type": "string",
                    "index": "not_analyzed",
                    "copy_to": "filename" 
                },
                "url": {
                    "type": "string",
                    "index": "not_analyzed",
                    "copy_to": "filename" 
                },
                "filename": {
                  "type": "string",
                  "analyzer": "urlanalyzer",
                  "store": false
                },
...

```

it comes back with no terms. There are no errors in the logs.

Also, if I try to use the reindex api to move data from another index into this one, it fails with the same error on every document:

```auto
            "cause": {
                "type": "illegal_argument_exception",
                "reason": "first position increment must be > 0 (got 0) for field 'filename'"
            },
            "status": 400

```

Is it because I'm calling `CharTermAttribute#append` with the wrong parameters? Here is my tokenizer:

```auto
public class UrlTokenizer extends Tokenizer {

    private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);

    protected List<String> tokens = new ArrayList<>();

    protected String stringToTokenize;

    protected int position = 0;
    ...

    @Override
    public boolean incrementToken() throws IOException {
        if (position >= tokens.size()) {
            return false;
        } else {
            String token = tokens.get(position);
            termAtt.setEmpty().append(token, stringToTokenize.indexOf(token), token.length());
            position++;
            return true;
        }
    }

    final char[] buffer = new char[8192];
    private void fillBuffer(Reader input) throws IOException {
        int len;
        StringBuilder str = new StringBuilder();
        str.setLength(0);
        while ((len = input.read(buffer)) > 0) {
            str.append(buffer, 0, len);
        }
        stringToTokenize = str.toString();
        tokens = splitUrl(stringToTokenize); //Method to split into segmentation permutations
    }

}

```

Can anyone tell me what I'm doing wrong?

Thanks for any help at all!

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [September 25, 2017, 2:05pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/2 "2017-09-25T14:05:44Z")

</div>

Here is the full class in case that helps (be gentle - I've been hacking this for _days_ trying to use an undocumented API):

```auto
public class UrlTokenizer extends Tokenizer {

    private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);

    protected List<String> tokens = new ArrayList<>();

    protected String stringToTokenize;

    protected int position = 0;

    public UrlTokenizer() {
        super(DEFAULT_TOKEN_ATTRIBUTE_FACTORY);
    }

    @Override
    public boolean incrementToken() throws IOException {
        if (position >= tokens.size()) {
            return false;
        } else {
            String token = tokens.get(position);
            termAtt.setEmpty().append(token, stringToTokenize.indexOf(token), token.length());
            position++;
            return true;
        }
    }

    private List<String> splitUrl(String url) throws MalformedURLException {
        URL realUrl = new URL(url);
        Set<String> parts = new HashSet<>();
        String host = realUrl.getHost();
        parts.add(host);

        String query = realUrl.getQuery();
        if (query != null && !"".equals(query.trim())) {
            parts.addAll(Arrays.asList(realUrl.getQuery().split("&")));
            parts.add(realUrl.getQuery());
            parts.add(url.substring(0, url.length() - (query.length() + 1)));
        }

        String path = realUrl.getPath();
        if (path != null && !"".equals(path.trim())) {
            parts.add(realUrl.getPath());
            String[] pathSegments = realUrl.getPath().substring(1).split("/");

            int idx = url.indexOf(realUrl.getHost() + "/") + (realUrl.getHost() + "/").length();
            String upToHost = url.substring(0, idx);

            for (int i = 0; i < pathSegments.length; i++) {
                parts.add(pathSegments[i]);
                for (int j = i+1; j <= pathSegments.length; j++) {
                    String permutation = join(Arrays.copyOfRange(pathSegments, i, j), "/"); // + (j == pathSegments.length ? "" : "/");
                    parts.add(/* "/" + */permutation);
                    if (i == 0) {
                        parts.add(upToHost + permutation);
                        parts.add(host + "/" + permutation);
                    }

                }

            }

        }

        return new ArrayList<>(parts);
    }

    public String join(String[] values, String delimiter) {
        StringBuffer strbuf = new StringBuffer();

        boolean first = true;

        for (String value : values) {
            if (!first) { strbuf.append(delimiter); } else { first = false; }
            strbuf.append(value);
        }

        return strbuf.toString();
    }
//
// @Override
// public void end() throws IOException {
// super.end();
// final int ofs = correctOffset(tokens.size());
// offsetAtt.setOffset(ofs, ofs);
// }

    @Override
    public void close() throws IOException {
        try {
            super.close();
        } finally {
            tokens.clear();
        }
    }

    @Override
    public void reset() throws IOException {
        super.reset();
        fillBuffer(input);
        position = 0;
    }

    final char[] buffer = new char[8192];
    private void fillBuffer(Reader input) throws IOException {
        int len;
        StringBuilder str = new StringBuilder();
        str.setLength(0);
        while ((len = input.read(buffer)) > 0) {
            str.append(buffer, 0, len);
        }
        stringToTokenize = str.toString();
        tokens = splitUrl(stringToTokenize);
    }

}

```

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [September 25, 2017, 6:37pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/3 "2017-09-25T18:37:24Z")

</div>

A couple things I notice:

- You don't need to call the `append` method that takes start/end, since you already have the entire token.
- You need to call `clearAttributes()` in the first call to `incrementToken()` (before setting any attribute values).
- Your offsets are all 0 because you don't have a start/end offset attribute.

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [September 25, 2017, 7:05pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/4 "2017-09-25T19:05:57Z")

</div>

Awesome - thanks!  
So, do I even need to set start/end offset attributes?

I have tried adding this:

```auto
    private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
...
            int startOffset = stringToTokenize.indexOf(token);
            offsetAtt.setOffset(startOffset, startOffset + token.length());

```

but am getting this error:

```auto
Caused by: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=-1,endOffset=29

```

I think this is because the field I am tokenizing is created by `copy_to` from two other fields. I think the `indexOf` is returning -1 because it's looking for a token from one of the fields in the other field's content?

Also, just to confirm - do you mean to say that I should call `clearAttributes()` every time `incrementToken` is called? Or just the first time?

Thanks again!

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [September 25, 2017, 8:00pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/5 "2017-09-25T20:00:25Z")

</div>

> [@ndtreviv](#):
>
> do you mean to say that I should call clearAttributes() every time incrementToken is called? Or just the first time?

Every time. See for example PatternTokenizer in lucene, and also the javadocs on Tokenizer.

> do I even need to set start/end offset attributes?

It depends. If you don't need offsets (for example, to do highlighting), then no.

> I think this is because the field I am tokenizing is created by copy\_to from two other fields. I think the indexOf is returning -1 because it's looking for a token from one of the fields in the other field's content?

I'm not sure what you mean by this. The offsets should be based on the data coming into the tokenizer. A copy\_to copies the data, and is handled outside of the analyzer, so a tokenizer should see no difference between values passed directly to a field, or copied from another field. I also would not use `indexOf` anyways: it has linear time lookup. If you want offsets, keep track of them when doing your splitting.

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [September 25, 2017, 8:52pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/6 "2017-09-25T20:52:34Z")

</div>

Great advice. Thank you!

So, I reindexed 10m documents into an index that used my tokenizer. Unfortunately only the `url` field terms made it across, not the `source` field terms.

Both fields are `copy_to`-d into the `filename` field:

```auto
...
               "source": {
                    "type": "string",
                    "index": "not_analyzed",
                    "copy_to": "filename" 
                },
                "url": {
                    "type": "string",
                    "index": "not_analyzed",
                    "copy_to": "filename" 
                },
                "filename": {
                  "type": "string",
                  "analyzer": "urlanalyzer",
                  "store": false
                },
...

```

I'm getting the string to tokenize in the `reset` method:

```auto
    @Override
    public void reset() throws IOException {
        super.reset();
        fillBuffer(input);
        position = 0;
    }

    final char[] buffer = new char[8192];
    private void fillBuffer(Reader input) throws IOException {
        int len;
        StringBuilder str = new StringBuilder();
        str.setLength(0);
        while ((len = input.read(buffer)) > 0) {
            str.append(buffer, 0, len);
        }
        stringToTokenize = str.toString();
        tokens = splitUrl(stringToTokenize);
    }

```

and clearing the tokens in the `close` method:

```auto
    @Override
    public void close() throws IOException {
        try {
            super.close();
        } finally {
            tokens.clear();
        }
    }

```

but not really doing anything with `end`.  
My guess is that I need to clear the tokens in `end`, not `close`...or set position back to 0 on `close` not `reset`.

Man, I really wish I understood how this worked!

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [September 25, 2017, 9:28pm UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/7 "2017-09-25T21:28:17Z")

</div>

You should read the Lucene [docs for analysis components](https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/analysis/package-summary.html).

And probably also the [TokenStream docs](https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/analysis/TokenStream.html), which Tokenizer implements.

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [September 26, 2017, 8:41am UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/8 "2017-09-26T08:41:20Z")

</div>

That's really great - thanks!

It turns out that I had created the index with an old template, so the `source` field wasn't being `copy_to`-d after all. D'oh!

Thanks again for your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 24, 2017, 8:41am UTC](https://discuss.elastic.co/t/custom-tokenizer-doesnt-work-on-reindex-index-api-only--analyze-endpoint/101715/9 "2017-10-24T08:41:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.