# Elasticsearch filter comparison with "preserve\_original": true

**URL:** <https://discuss.elastic.co/t/elasticsearch-filter-comparison-with-preserve-original-true/356468>\
**Category:** Elasticsearch\
**Created:** [March 29, 2024, 5:30pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-comparison-with-preserve-original-true/356468 "2024-03-29T17:30:35Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nkachami](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@nkachami](https://discuss.elastic.co/u/nkachami)\
**Post date:** [March 29, 2024, 5:30pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-comparison-with-preserve-original-true/356468/1 "2024-03-29T17:30:35Z")

</div>

Hello,

I am testing the outputs of different analyzer configurations and found one that does not seem to make much sense.

Lets imagine we have two settings blocks:

```auto
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default": {
          "tokenizer": "standard",
          "filter": ["lowercase", "ascii_folding_original_preserving"]
        },
      },
      "filter": {
        "ascii_folding_original_preserving": {
          "type": "asciifolding",
          "preserve_original": true
        }
      }
    }
  }
}

```

and

```auto
{
  "settings": {
    "analysis": {
      "analyzer": {
        "default": {
          "tokenizer": "icu_tokenizer",
          "filter": ["lowercase", "icu_folding_original_preserving"]
        },
      },
      "filter": {
        "icu_folding_original_preserving": {
          "type": "icu_folding",
          "preserve_original": true
        }
      }
    }
  }
}

```

Lets assume we run both of these analyze API requests with full width characters:

```auto
GET /_analyze
{
  "tokenizer": "standard",
  "filter": ["lowercase", "ascii_folding_original_preserving"],
  "text":"Ｃｕｌｔｕｒｅ ｏｆ Ｊａｐａｎ"
}

GET /_analyze
{
  "tokenizer": "icu_tokenizer",
  "filter": ["lowercase", "icu_folding_original_preserving"],
  "text":"Ｃｕｌｔｕｒｅ ｏｆ Ｊａｐａｎ"
}

```

Based on the [documentation](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-folding.html) the icu\_folding filter is a "Case folding of Unicode characters based on `UTR#30` , like the [ASCII-folding token filter](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/analysis-asciifolding-tokenfilter.html) on steroids."

With that being said, I expected the same exact response with the correct one being the 1st analysis result below but instead received:

1st analysis result with ascii\_folding\_original\_preserving filter

```auto
{
  "tokens" : [
    {
      "token" : "culture",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "ｃｕｌｔｕｒｅ",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "of",
      "start_offset" : 8,
      "end_offset" : 10,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "ｏｆ",
      "start_offset" : 8,
      "end_offset" : 10,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "japan",
      "start_offset" : 11,
      "end_offset" : 16,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "ｊａｐａｎ",
      "start_offset" : 11,
      "end_offset" : 16,
      "type" : "<ALPHANUM>",
      "position" : 2
    }
  ]
}

```

2nd analysis result with icu\_folding\_original\_preserving filter:

```auto
{
  "tokens" : [
    {
      "token" : "culture",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "of",
      "start_offset" : 8,
      "end_offset" : 10,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "japan",
      "start_offset" : 11,
      "end_offset" : 16,
      "type" : "<ALPHANUM>",
      "position" : 2
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 26, 2024, 5:31pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-comparison-with-preserve-original-true/356468/2 "2024-04-26T17:31:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.