# Issue with greek language

**URL:** <https://discuss.elastic.co/t/issue-with-greek-language/90535>\
**Category:** Elasticsearch\
**Created:** [June 22, 2017, 8:18pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535 "2017-06-22T20:18:25Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 22, 2017, 8:18pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/1 "2017-06-22T20:18:25Z")

</div>

Hello,

i try to resolve the issue with search the names on greek. if a greek name in index is uppercase and search is done with lowercase, then i don't receive any results. however, if the same scenario applies to english names then the search works well. i try to use such custom analyzer:

```
                analyzer: {
                      greek_analyzer: {
                        type : custom,
                        tokenizer : standard,
                        filter : [my_greek_lowercase]
                      }
                    },

                   filter : { 
                        my_greek_lowercase : {
                            type : lowercase
                            name : greek
                        }
                    }
            }

```

and i have such mapping:

```
        fullName:{type:string, analyzer:some_my_default_analyzer, 
                          fields: {raw:{type: String, index: not_analyzed}, 
                                       greek: { type: string, analyzer: greek_analyzer }}}

```

do you have any ideas what could be wrong with this?

thank you!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 22, 2017, 10:20pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/2 "2017-06-22T22:20:04Z")

</div>

Could you provide a full recreation script as described in

> [@About the Elasticsearch category](https://discuss.elastic.co/t/about-the-elasticsearch-category/21):
>
> The heart of the free and open Elastic Stack Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease. warning PLEASE READ THIS SECTION IF IT'S YOUR FIRST POST Some useful links: [elasticsearch reference guide](http://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)[elasticsearch user guide](http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html)[elasticsearch plugins](https://www.elastic.co/guide/en/elasticsearch/plugins/current/index.html)[elasticsearch cl…](https://www.elastic.co/guide/en/elasticsearch/client/index.html)

It will help to better understand what you are doing.  
Please, try to keep the example as simple as possible.

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 23, 2017, 1:17am UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/3 "2017-06-23T01:17:38Z")

</div>

Here is the full mapping:

```
{
settings:{
    index: {
        analysis :{                    
            analyzer: {
              greek_analyzer: {
                type : custom,
                tokenizer : standard,
                filter : [my_greek_lowercase]
              },
              skipVerbs: {
                 type : custom,
                 tokenizer : standard,
                 filter : [standard, lowercase]
                }
            },
            filter : { 
                my_greek_lowercase : {
                    type : lowercase
                    name : greek
                }
            }
        }
    }
},
mappings : {
    user: {
       properties:{
            fullName:{
                type:string, analyzer:skipVerbs, 
                fields: {
                    raw:{type: String, index: not_analyzed}, 
                    greek: { type: string, analyzer: greek_analyzer }
                }
            }
       }
    }
}
 }

```

the value of fullName in the index could be 'ΝΙΚΟΣ' but search for that value could be done in lowercase 'νικος', what is the same name. If we are talking about english values then there is no problem, f.e. value in index is JOHN but i search by john and i can find that name.

Could you give me your opinion about it?

Thank you!

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 23, 2017, 1:42am UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/4 "2017-06-23T01:42:55Z")

</div>

this is query i use for search:

```
                   {"query":{  
  "filtered":{  
     "query":{  
        "bool":{  
           "should":[  
              {  
                 "wildcard":{  
                    "fullName":"*νικος*"
                 }
              }
           ],
           "minimum_should_match":1
        }
     }
  }},"version":true
      }
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 23, 2017, 7:01am UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/5 "2017-06-23T07:01:04Z")

</div>

Could you provide a full script that I can just run on 5.4.2?

And yeah, use 5.x if possible.

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 23, 2017, 12:14pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/6 "2017-06-23T12:14:58Z")

</div>

What do you mean the full script? If i understand you correctly, we are using Java to create/update index and we define there mapping and according to that mapping we are updating it by using JestClient. I have to check with our devops about upgrading it. currently, we are using version 2.3. could it be too old?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 23, 2017, 4:28pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/7 "2017-06-23T16:28:39Z")

</div>

I mean a script I can paste in Kibana console and play with.

As explained in the very first post I linked to.

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 23, 2017, 6:51pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/8 "2017-06-23T18:51:07Z")

</div>

I see.

this i use to put a greek name in uppercase:

```
 PUT index/type/1
{
  "fullName": "ΝΙΚΟΣ"
}

```

here i try to get this name by using lowercase:

```
POST index/type/_search
{"query":{  
      "filtered":{  
         "query":{  
            "bool":{  
               "should":[  
                  {  
                     "wildcard":{  
                        "fullName":"*νικος*"
                     }
                  }
               ],
               "minimum_should_match":1
            }
         }
      }},
    "version":true
    }

```

Sorry, I never used Kibana... Let me, please, know whether you need something more to reproduce it?

Thanks much for help!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 26, 2017, 2:33pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/9 "2017-06-26T14:33:50Z")

</div>

Knowing that wildcard queries are not analyzed, for sure `νικος` does not match `ΝΙΚΟΣ`. Not sure BTW what `ΝΙΚΟΣ` is rendered to when using a standard analyzer.

If you are using a specific analyzer and mapping, please provide them within the script.

A script is something like:

```auto
DELETE index
PUT index
{
  // Index settings and mapping here
}
PUT index/doc/1
{
  "foo": "bar"
}
GET index/_search
{
  "query": {
    "match": {
      "foo": "bar"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 26, 2017, 3:01pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/10 "2017-06-26T15:01:43Z")

</div>

Yes, I'm using specific analyzer which was working well for english(it is called 'skipVerbs'). once we needed to add a greek we got mentioned issue and i tried to play around with additional analyzer for greek language but no luck with it.

```
PUT some_index
{
"settings":{
    "index": {
        "analysis" :{                    
            "analyzer": {
              "greek_analyzer": {
                "type" : "custom",
                "tokenizer" : "standard",
                "filter" : ["my_greek_lowercase"]
              },
              "skipVerbs": {
                 "type" : "custom",
                 "tokenizer" : "standard",
                 "filter" : ["standard", "lowercase"]
                }
            },
            "filter" : { 
                "my_greek_lowercase" : {
                    "type" : "lowercase",
                    "name" : "greek"
                }
            }
        }
    }
},
"mappings" : {
    "user": {
       "properties":{
            "fullName":{
                "type": "string", 
                "analyzer":"skipVerbs", 
                "fields": {
                    "raw":{"type": "String", "index": "not_analyzed"}, 
                    "greek": { "type": "string", "analyzer": "greek_analyzer" }
                }
            }
       }
    }
}
 }

PUT some_index/some_type/1
{
  "fullName": "ΝΙΚΟΣ"
}

POST some_index/some_type/_search
{
	"query": {
		"filtered":{
			"query": {
				"bool":{
					"should":[{
						"wildcard":{ 
							"fullName" : "*νικος*" }
							} ],
							"minimum_should_match" : 1
				}
			}
		}
	},
	"version":true
}

```

Thanks!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 26, 2017, 4:26pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/11 "2017-06-26T16:26:31Z")

</div>

If you use the `_analyze` API , it will tell you how your string is actually indexed.

```auto
DELETE some_index
PUT some_index
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "greek_analyzer": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": [
              "my_greek_lowercase"
            ]
          },
          "skipVerbs": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": [
              "standard",
              "lowercase"
            ]
          }
        },
        "filter": {
          "my_greek_lowercase": {
            "type": "lowercase",
            "name": "greek"
          }
        }
      }
    }
  },
  "mappings": {
    "doc": {
      "properties": {
        "fullName": {
          "type": "text",
          "analyzer": "skipVerbs",
          "fields": {
            "raw": {
              "type": "keyword"
            },
            "greek": {
              "type": "text",
              "analyzer": "greek_analyzer"
            }
          }
        }
      }
    }
  }
}
POST some_index/_analyze
{
  "text": "ΝΙΚΟΣ",
  "analyzer": "greek_analyzer"
}

```

This gives:

```auto
{
  "tokens": [
    {
      "token": "νικοσ",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    }
  ]
}

```

`νικοσ` is not `νικος` (I can't comment on what it means in greek though). 😃

Which means that the following match:

```auto
PUT some_index/doc/1
{
  "fullName": "ΝΙΚΟΣ"
}

POST some_index/doc/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "wildcard": {
            "fullName.greek": "*νικοσ*"
          }
        }
      ]
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 26, 2017, 9:39pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/12 "2017-06-26T21:39:04Z")

</div>

I see. I didn't know about \_analyze API... It seems like Java and elasticsearch differently convert words to lowercase...

Thanks David!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 28, 2017, 9:14am UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/13 "2017-06-28T09:14:38Z")

</div>

> [@olehd](#):
>
> It seems like Java and elasticsearch differently convert words to lowercase...

Hmmm. Interesting (if I understood correctly what you meant).  
I wonder if you have an encoding issue in your java Application. Are you using UTF-8?

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 28, 2017, 3:42pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/14 "2017-06-28T15:42:21Z")

</div>

Yes, we use UTF-8.  
You can try to lowercase this name 'ΝΙΚΟΣ' on your machine and to compare the results. but, yes, the issue is very interesting. pity, that i have no time to investigate it more...  
if you have a chance and wish to take a look into it, please, let me know the results 🙂

Thanks!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 28, 2017, 4:10pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/15 "2017-06-28T16:10:32Z")

</div>

I looked at the Lucene source code.

The LowerCase filter does that: [https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/analysis/LowerCaseFilter.java#L42-L49](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/analysis/LowerCaseFilter.java#L42-L49)

```auto
  @Override
  public final boolean incrementToken() throws IOException {
    if (input.incrementToken()) {
      CharacterUtils.toLowerCase(termAtt.buffer(), 0, termAtt.length());
      return true;
    } else
      return false;
  }

```

Which calls: [https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/analysis/CharacterUtils.java#L47-L62](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/analysis/CharacterUtils.java#L47-L62)

```auto
  /**
   * Converts each unicode codepoint to lowerCase via {@link Character#toLowerCase(int)} starting 
   * at the given offset.
   * @param buffer the char buffer to lowercase
   * @param offset the offset to start at
   * @param limit the max char in the buffer to lower case
   */
  public static void toLowerCase(final char[] buffer, final int offset, final int limit) {
    assert buffer.length >= limit;
    assert offset <=0 && offset <= buffer.length;
    for (int i = offset; i < limit;) {
      i += Character.toChars(
              Character.toLowerCase(
                  Character.codePointAt(buffer, i, limit)), buffer, i);
     }
  }

```

Which calls the java.util classes then.

Indeed, `"ΝΙΚΟΣ".toLowerCase()` produces a different result: `νικος`:

```auto
String text = "ΝΙΚΟΣ";
for (int i = 0; i < text.length(); i++) {
    char c = Character.toLowerCase(text.charAt(i));
}

```

This produces `νικοσ`.

Anyone speaking greek can tell which form is correct or wrong?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 28, 2017, 4:33pm UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/16 "2017-06-28T16:33:13Z")

</div>

I spoke with @dliappis about this and actually the real lowercase form should be `νίκος`.

Probably for greek language you should use instead the analysis-icu plugin.  
Specifically the [https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-transform.html](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-transform.html)

Apparently (I have never used it), you can do lowercase transformation, as explained in [http://userguide.icu-project.org/transforms/general](http://userguide.icu-project.org/transforms/general) with `Any-Lower`.

HTH.

---

<div class="post-metadata">

**Author:** ![olehd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olehd/32/19452_2.png) [@olehd](https://discuss.elastic.co/u/olehd)\
**Post date:** [June 29, 2017, 12:06am UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/17 "2017-06-29T00:06:01Z")

</div>

i didn't play with analyzers but i did the fix by using java. i have just introduced new property in the index with lowercase value and i convert this value to lowercase by using java(no need to use an elastic search analyzers). my fix should resolve an issue with other similar languages in future.

thanks a lot for help!  
i appreciate it a lot!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 27, 2017, 12:06am UTC](https://discuss.elastic.co/t/issue-with-greek-language/90535/18 "2017-07-27T00:06:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.