# Word delimiter filter with preserve\_original

**URL:** <https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123>\
**Category:** Elasticsearch\
**Created:** [November 8, 2019, 1:59pm UTC](https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123 "2019-11-08T13:59:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![skowron-line](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/skowron-line/32/43528_2.png) [@skowron-line](https://discuss.elastic.co/u/skowron-line)\
**Post date:** [November 8, 2019, 1:59pm UTC](https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123/1 "2019-11-08T13:59:59Z")

</div>

I have such case  
My mapping is

```
PUT my_index
{
  "settings": {
    "analysis": {
      "filter": {
        "my_length": {
          "type": "length",
          "min": 2
        },
        "my_word": {
          "type": "word_delimiter",
          "preserve_original": true
        }
      },
      "analyzer": {
        "std_folded": { 
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "asciifolding",
            "my_word",
            "my_length"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "my_text": {
        "type": "text",
        "analyzer": "std_folded" 
      }
    }
  }
}

```

And Im trying to analyze string "W.100"

```
GET my_index/_analyze 
{
  "analyzer": "std_folded", 
  "text": ["W.1000"]
}

```

and Im getting

```
{
  "tokens" : [
    {
      "token" : "1000",
      "start_offset" : 2,
      "end_offset" : 6,
      "type" : "<NUM>",
      "position" : 1
    }
  ]
}

```

The question is why I am not receiving also W.1000 token ?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 8, 2019, 4:10pm UTC](https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123/2 "2019-11-08T16:10:04Z")

</div>

Hey,

without debugging too much I think this could be caused by the `standard` tokenizer. Try this

```auto
GET _analyze
{
  "text": [
    "W.1000"
  ],
  "tokenizer": "keyword",
  "filter": [
    { "type" : "lowercase" }, 
    { "type" : "asciifolding" },
    { "type": "length", "min": 2 },
    { "type" : "word_delimiter", "preserve_original": true }
  ]
}

```

Also, try switching the `length` and `word_delimiter` filters around for different results. However the above would only work, if w.1000 was the only string in your field, which I am not sure, it is?

The standard tokenizer would already have tokenized and potentally removed the `w.1000` even before the word delimiter filter would be able to kick in. You could try playing around with a different tokenizer like `whitespace` or `classic` and see if that helps as well.

---

<div class="post-metadata">

**Author:** ![skowron-line](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/skowron-line/32/43528_2.png) [@skowron-line](https://discuss.elastic.co/u/skowron-line)\
**Post date:** [November 12, 2019, 6:21am UTC](https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123/3 "2019-11-12T06:21:05Z")

</div>

The output of Your solution is

```
{
  "tokens" : [
    {
      "token" : "w.1000",
      "start_offset" : 0,
      "end_offset" : 6,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "w",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "1000",
      "start_offset" : 2,
      "end_offset" : 6,
      "type" : "word",
      "position" : 1
    }
  ]
}

```

and I dont want `w` token, only 2 and more token lengths

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 20, 2019, 3:05pm UTC](https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123/4 "2019-11-20T15:05:47Z")

</div>

have you tried changing the order of the filters?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 18, 2019, 3:05pm UTC](https://discuss.elastic.co/t/word-delimiter-filter-with-preserve-original/207123/5 "2019-12-18T15:05:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.