Case insensitive sorting problem

[djmcgreal (Daniel McGreal)]

Hi,
I'd like to use the ducet sorting available from the icu plugin but am finding it difficult. Is there a definitive example somewhere? There's problematic differences between the examples in the definitive guide (2.x), the documentation of 5.2 (the version I'm using) and the latest thinking.

I'm using the following mapping
{
"template": "itemit-state-",
"mappings": {
"default" : {
"dynamic_templates" : [
{
"not_analyzed" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above": 256
},
"ducet_sort" : {
"type" : "keyword",
"analyzer" : "ducet_sort"
}
}
}
}
}
]
}
},
"settings": {
"index.mapping.ignore_malformed": true,
"analysis": {
"analyzer": {
"ducet_sort": {
"tokenizer": "keyword",
"filter": [ "icu_collation" ]
}
}
}
}
}

But I get: java.lang.IllegalArgumentException: Custom Analyzer [ducet_sort] failed to find filter under name [icu_collation]

Something to do with normalisers?

Thanks, Dan.

[djmcgreal (Daniel McGreal)]

Ok so I'm now in a position to better phrase the problem!

Following https://www.elastic.co/guide/en/elasticsearch/plugins/5.2/analysis-icu-collation.html works, but involves enabling field data for the field. I was hoping to avoid this and use keyword instead, but as soon as I do that I get an error at index time:

Mapping definition for [fields] has unsupported parameters: [analyzer : ducet_sort]

Any help appreciated.
Best, Dan.

[abdon (Abdon Pijpelink)]

You cannot apply an analyzer to a field of type keyword. Only text fields are analyzed. You have to use an normalizer instead. Unfortunately, I don't think you can use the icu_collation filter in a normalizer.

The best solution is to upgrade to version 5.5, which has a brand new icu_collation_keyword field type. The docs have more info: https://www.elastic.co/guide/en/elasticsearch/plugins/5.5/analysis-icu-collation-keyword-field.html

If you cannot upgrade to 5.5, the only solution is to use a text field instead of a keyword field and apply your analyzer to that text field. The downside is that you will have to enable fielddata if you decide to go that way.

[djmcgreal (Daniel McGreal)]

Thanks for the clarification! This is indeed the realisation we reached…. We’ll bite the fielddata until such time as we can upgrade…
Thanks, Dan.