# Case insensitive sort doesn't work

**URL:** https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192
**Category:** Elasticsearch
**Created:** [August 6, 2018, 4:36pm UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192 "2018-08-06T16:36:06Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![shwetaskatdare](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@shwetaskatdare](https://discuss.elastic.co/u/shwetaskatdare)
#### Post date: [August 6, 2018, 4:36pm UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/1 "2018-08-06T16:36:06Z")

</div>

**Elasticsearch version** (`bin/elasticsearch --version`): 5.3

**JVM version** (`java -version`): java 10.0.1 2018-04-17

**Description of the problem including expected versus actual behavior** :  
Hello,  
I am trying to achieve case insensitive sort using lowercase analyzer and keyword field. However, the result is not as expected. I am hesitant to use normalizers because of it being an experimental feature. I've tried using default analyzer and also explicitly setting `standard` analyzer. There was a similar issue [https://github.com/elastic/elasticsearch/issues/22410](https://github.com/elastic/elasticsearch/issues/22410), but it wasn't much help. Can you please help clarify if what I expect in the results is the correct ES behavior? if not, how can I correctly achieve case insensitive sorting?

Appreciate any help.

**Steps to reproduce** :

1. Create Index

```auto
PUT http://localhost:9200/testindex
{
  "settings": {
    "analysis": {
      "analyzer": {
        "case_insensitive": {
               "tokenizer": "lowercase"
        	}
    	}
    }
  }
}

```

1. Mapping

```auto
PUT http://localhost:9200/testindex/_mapping/testmapping
{
    "properties": {
      "Id": {
        "type": "keyword"
      },
      "Name": {
        "type": "text",
        "analyzer": "case_insensitive",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      }
    }
}

```

1. Documents

```auto
PUT http://localhost:9200/testindex/testmapping/1
{
	"Id": 1,
	"Name": "III-bbb"
}

PUT http://localhost:9200/testindex/testmapping/8
{
	"Id": 8,
	"Name": "III-ccc"
}

PUT http://localhost:9200/testindex/testmapping/2
{
	"Id": 2,
	"Name": "iii-aaa"
}

```

1. Search query

```auto
POST http://localhost:9200/testindex/testmapping/_search
{
	"query": {
		"match": {
			"Name": "iii"
		}
	},
	"sort": [
		{
			"Name.keyword": {
				"order": "asc"
			}
		}]
}

```

1. Actual search result

```auto
{
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 3,
        "max_score": null,
        "hits": [
            {
                "_index": "testindex",
                "_type": "testmapping",
                "_id": "1",
                "_score": null,
                "_source": {
                    "Id": 1,
                    "Name": "III-bbb"
                },
                "sort": [
                    "III-bbb"
                ]
            },
            {
                "_index": "testindex",
                "_type": "testmapping",
                "_id": "8",
                "_score": null,
                "_source": {
                    "Id": 8,
                    "Name": "III-ccc"
                },
                "sort": [
                    "III-ccc"
                ]
            },
            {
                "_index": "testindex",
                "_type": "testmapping",
                "_id": "2",
                "_score": null,
                "_source": {
                    "Id": 2,
                    "Name": "iii-aaa"
                },
                "sort": [
                    "iii-aaa"
                ]
            }
        ]
    }
}

```

1. Expected result:  
Order: Id: 2, Id: 1, & Id: 8

Based on the `_analyze` API result the text is being tokenized in lowercase as follow, so the results should be ordered alphabetically and not lexicographically

```auto
POST http://localhost:9200/testindex/_analyze
{
	"field": "testmapping.Name",
	"text": "IIII-bbb"
}

Result: 
{
    "tokens": [
        {
            "token": "iiii",
            "start_offset": 0,
            "end_offset": 4,
            "type": "<ALPHANUM>",
            "position": 0
        },
        {
            "token": "bbb",
            "start_offset": 5,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 1
        }
    ]
}

```

---

<div class="post-metadata">

### Author: ![nourtemp](https://avatars.discourse-cdn.com/v4/letter/n/bc79bd/32.png) [@nourtemp](https://discuss.elastic.co/u/nourtemp)
#### Post date: [August 11, 2018, 7:37pm UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/2 "2018-08-11T19:37:32Z")

</div>

Just following this question, as I have the exact problem.

---

<div class="post-metadata">

### Author: ![klof](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klof/32/31590_2.png) [@klof](https://discuss.elastic.co/u/klof)
#### Post date: [August 12, 2018, 6:28am UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/3 "2018-08-12T06:28:20Z")

</div>

The `keyword` version of a field is not analyzed, meaning you will be able to search, sort and aggregate only on exact values.  
Your logic implies to sort on the analyzed version of the field name (`Name`), but when you sort or aggregate on an analyzed field, you get an error message, that requires the field to enable **fielddata** :

[https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#\_enabling\_fielddata\_on\_literal\_text\_literal\_fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#_enabling_fielddata_on_literal_text_literal_fields)

```
PUT testindex/_mapping/testmapping
{
  "properties": {
    "Id": {
      "type": "keyword"
    },
    "Name": {
      "type": "text",
      "analyzer": "case_insensitive",
      "fielddata": true
    }
  }
}

```

Then, your search with the sort on the analyzed field will give you the expected result :

```
POST testindex/testmapping/_search
{
  "query": {
    "match": {
      "Name": "iii"
    }
  },
  "sort": [
    {
      "Name": {
        "order": "asc"
      }
    }
  ]
}

```

However, be aware that enable fielddata can be expensive, as mentioned in the documentation, and maybe it would be better to modify your documents ( **create a new field "name\_lowercase", with the keyword type** ) then sort on this field.

[https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#\_fielddata\_is\_disabled\_on\_literal\_text\_literal\_fields\_by\_default](https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#_fielddata_is_disabled_on_literal_text_literal_fields_by_default)

---

<div class="post-metadata">

### Author: ![nourtemp](https://avatars.discourse-cdn.com/v4/letter/n/bc79bd/32.png) [@nourtemp](https://discuss.elastic.co/u/nourtemp)
#### Post date: [August 12, 2018, 2:31pm UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/4 "2018-08-12T14:31:56Z")

</div>

> [@klof](#):
>
> The `keyword` version of a field is not analyzed, meaning you will be able to search, sort and aggregate only on exact values.

That's actually what I want (Assuming is the same what the question owner wants).  
I want to sort based on the exact values, not the analyzed ones.  
That's why I sort using "Name.keyword" not "Name" which I believe the correct way to do the non-expensive sorting.

If not, what is the optimum way to sort case-insensitively on a text field (like Person Full Name) ?

Right?

---

<div class="post-metadata">

### Author: ![klof](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klof/32/31590_2.png) [@klof](https://discuss.elastic.co/u/klof)
#### Post date: [August 13, 2018, 5:47am UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/5 "2018-08-13T05:47:24Z")

</div>

> [@nourtemp](#):
>
> That's actually what I want (Assuming is the same what the question owner wants).

I'm not sure, i understood he wanted to sort, the analyzed values (names in lowercase).

However, you're right, it's better to sort on keyword values (case-sensitive), but be aware that uppercase letters will be sorted before lowercase letters.  
You can debug it with your programming language. if you sort an array with values :

```
["a", "b", "B", "C", "c", "A"] 
the result is : 
["A", "B", "C", "a", "b", "c"]

```

In your case if your data are properly formatted there's no issue to sort by keyword, otherwise I think you should create a new field (name\_lowercase) just for this purpose.

---

<div class="post-metadata">

### Author: ![shwetaskatdare](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@shwetaskatdare](https://discuss.elastic.co/u/shwetaskatdare)
#### Post date: [August 16, 2018, 5:03pm UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/6 "2018-08-16T17:03:44Z")

</div>

thanks for your input @klof and @nourtemp. I want to sort on the not analyzed values (or exact value) which is why I am sorting on `Name.keyword` (if you see the query). I do not want to enable the fielddata because of what's mentioned in the document. Looking for a non expensive way of sorting

---

<div class="post-metadata">

### Author: ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)
#### Post date: [August 16, 2018, 9:18pm UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/7 "2018-08-16T21:18:03Z")

</div>

To sort case-insensitively on a keyword field, you can apply a [normalizer](https://www.elastic.co/guide/en/elasticsearch/reference/current/normalizer.html) to that field. You can think of normalizers as analyzers for keyword fields instead of text fields. So, instead of defining an analyzer in your settings, define a normalizer:

```auto
PUT /testindex
{
  "settings": {
    "analysis": {
      "normalizer": {
        "case_insensitive": {
          "filter": "lowercase"
        }
      }
    }
  }
}

```

Next, apply that normalizer to your keyword field in your mapping:

```auto
PUT /testindex/_mapping/testmapping
{
  "properties": {
    "Id": {
      "type": "keyword"
    },
    "Name": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "normalizer": "case_insensitive"
        }
      }
    }
  }
}

```

Now, you will be able to sort case-insensitively on `Name.keyword` without using fielddata.

---

<div class="post-metadata">

### Author: ![nourtemp](https://avatars.discourse-cdn.com/v4/letter/n/bc79bd/32.png) [@nourtemp](https://discuss.elastic.co/u/nourtemp)
#### Post date: [September 11, 2018, 10:48am UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/8 "2018-09-11T10:48:30Z")

</div>

> [@abdon](#):
>
> you can apply a [normalizer] to that field

Hello abdon  
Thanks for you reply.  
Actually, I'm using NEST as client as my project is .Net based.  
AFAI, NEST doesn't support normalizers yet.  
Correct me if I'm wrong.

---

<div class="post-metadata">

### Author: ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)
#### Post date: [September 11, 2018, 11:08am UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/9 "2018-09-11T11:08:30Z")

</div>

What problem are you running into specifically? It seems NEST added support for normalizers back in February 2017. Are you on an old version?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 9, 2018, 11:08am UTC](https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/10 "2018-10-09T11:08:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.