# Non-matched tokens not filtered: pattern\_capture with preserve\_original: false

**URL:** <https://discuss.elastic.co/t/non-matched-tokens-not-filtered-pattern-capture-with-preserve-original-false/40793>\
**Category:** Elasticsearch\
**Created:** [February 2, 2016, 10:40pm UTC](https://discuss.elastic.co/t/non-matched-tokens-not-filtered-pattern-capture-with-preserve-original-false/40793 "2016-02-02T22:40:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![haight6716](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/haight6716/32/7545_2.png) [@haight6716](https://discuss.elastic.co/u/haight6716)\
**Post date:** [February 2, 2016, 10:40pm UTC](https://discuss.elastic.co/t/non-matched-tokens-not-filtered-pattern-capture-with-preserve-original-false/40793/1 "2016-02-02T22:40:51Z")

</div>

I have a custom `pattern` tokenizer that creates tokens which include colon (`:`). I also have a custom `pattern_capture` token filter. I have `preserve_original` set to `false`.

The problem is that tokens which don't match any of my patterns are being passed through the filter.

Here's a simplified test case:

```
            "filter": {
                "ipnwords": {
                    "type": "pattern_capture",
                    "preserve_original": false,
                    "patterns": [
                        "([a-z0-9][a-z0-9-]+[a-z0-9])"
                    ]
                }
            },
            "analyzer": {
                "words": {
                    "tokenizer": "ipwords",
                    "filter": ["lowercase", "ipnwords", "unique"]
                }
            },
            "tokenizer": {
                "ipwords": {
                    "type": "pattern",
                    "pattern": "[a-zA-Z0-9:\\.-]+",
                    "group": 0
                }
            }

```

When I pass a test document to `/_analyze` that includes the string `:::`, that string is returned as a token, even though it should never match the pattern in the filter.

This seems like a bug (`preserve_orginal=false` ignored when no pattern matched in `pattern_capture`), but maybe someone can point out the error of my ways.

Thanks in advance for any help/advice. Let me know if I can provide more detail or help with testing/bugfix.

---

<div class="post-metadata">

**Author:** ![davidcai19840412](https://avatars.discourse-cdn.com/v4/letter/d/90db22/32.png) [@davidcai19840412](https://discuss.elastic.co/u/davidcai19840412)\
**Post date:** [August 24, 2016, 8:43am UTC](https://discuss.elastic.co/t/non-matched-tokens-not-filtered-pattern-capture-with-preserve-original-false/40793/2 "2016-08-24T08:43:58Z")

</div>

i have the same problem, and not sovled.

---

<div class="post-metadata">

**Author:** ![Blackeagle52](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackeagle52/32/16128_2.png) [@Blackeagle52](https://discuss.elastic.co/u/Blackeagle52)\
**Post date:** [March 6, 2017, 5:47pm UTC](https://discuss.elastic.co/t/non-matched-tokens-not-filtered-pattern-capture-with-preserve-original-false/40793/3 "2017-03-06T17:47:09Z")

</div>

Oké, had to create an account to reply onto this, but I had a similar problem and _maybe_ my solution could help you out. You helped me with the following sentence:

> [@haight6716](#):
>
> The problem is that tokens which don't match any of my patterns are being passed through the filter.

This made me understand what the tokenizer actually does.

But I think your problem lies in your custom Tokenizer, which has the pattern `"[a-zA-Z0-9:\\.-]+"`, but the docs say: "The default pattern is `\W+`, which splits text whenever it encounters non-word characters." Thus with your pattern you get a lot of tokens! You probably need to negate your pattern `"[^a-zA-Z0-9:\\.-]+"`.

To compare with my problem and solution, see [StackOverflow](http://stackoverflow.com/questions/42623381/fail-to-implement-pattern-capture-token-filter/)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:02pm UTC](https://discuss.elastic.co/t/non-matched-tokens-not-filtered-pattern-capture-with-preserve-original-false/40793/4 "2017-07-05T22:02:32Z")

</div>