# Russian words not work with synonym token filter
**URL:** https://discuss.elastic.co/t/russian-words-not-work-with-synonym-token-filter/16186
**Category:** Elasticsearch
**Created:** [March 6, 2014, 10:39am UTC](https://discuss.elastic.co/t/russian-words-not-work-with-synonym-token-filter/16186 "2014-03-06T10:39:51Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@marvin](https://discuss.elastic.co/u/marvin)
#### Post date: [March 6, 2014, 10:39am UTC](https://discuss.elastic.co/t/russian-words-not-work-with-synonym-token-filter/16186/1 "2014-03-06T10:39:51Z")
Hi.
I have test index with settings:
curl -XPOST '[http://localhost:9200/test\_index](http://localhost:9200/test_index)' -d '
{
"settings" : {
"number\_of\_shards" : 5,
"language":"javascript",
"analysis": {
"filter": {
"snowball\_text" : {
"type": "snowball",
"language": "Russian"
},
"synonym" : {
"type" : "synonym",
"synonyms\_path" : "synonym.txt"
}
},
"analyzer": {
"search" : {
"type" :"custom",
"tokenizer": "standard",
"filter": ["snowball\_text", "lowercase",
"russian\_morphology", "synonym"]
}
}
}
},
"mappings" : {
"test\_type" : {
"properties" : {
"test" : {
"type" : "string",
"analyzer" : "search"
},
"description" : {
"type" : "string",
"analyzer" : "search"
}
}
}
}
}'
File synonym.txt:
продажа =\> купить
аренда =\> арендовать, сниму, снять
foo =\> foo bar, baz
English words works fine:
curl -XGET
'[http://localhost:9200/test\_index/\_analyze?text=foo&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=foo&analyzer=search&pretty=true)'
{
"tokens" : [ {
"token" : "foo",
"start\_offset" : 0,
"end\_offset" : 3,
"type" : "SYNONYM",
"position" : 1
}, {
"token" : "baz",
"start\_offset" : 0,
"end\_offset" : 3,
"type" : "SYNONYM",
"position" : 1
}, {
"token" : "bar",
"start\_offset" : 0,
"end\_offset" : 3,
"type" : "SYNONYM",
"position" : 2
} ]
}
But russian:
curl -XGET
'[http://localhost:9200/test\_index/\_analyze?text=продажа&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=%D0%BF%D1%80%D0%BE%D0%B4%D0%B0%D0%B6%D0%B0&analyzer=search&pretty=true)'
{
"tokens" : [ {
"token" : "ﾀ",
"start\_offset" : 3,
"end\_offset" : 4,
"type" : "",
"position" : 1
}, {
"token" : "ﾾ",
"start\_offset" : 5,
"end\_offset" : 6,
"type" : "",
"position" : 2
}, {
"token" : "ﾴ",
"start\_offset" : 7,
"end\_offset" : 8,
"type" : "",
"position" : 3
}, {
"token" : "ﾰ",
"start\_offset" : 9,
"end\_offset" : 10,
"type" : "",
"position" : 4
}, {
"token" : "ﾶ",
"start\_offset" : 11,
"end\_offset" : 12,
"type" : "",
"position" : 5
}, {
"token" : "ﾰ",
"start\_offset" : 13,
"end\_offset" : 14,
"type" : "",
"position" : 6
} ]
}
I cant't understand what i'm doing wrong?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [March 6, 2014, 5:31pm UTC](https://discuss.elastic.co/t/russian-words-not-work-with-synonym-token-filter/16186/2 "2014-03-06T17:31:41Z")
Despite my name, I do not speak Russian. 🙂 Please excuse my ignorance of
the Russian language while I attempt to debug.
Currently, the synonym token filter is being applied after the other three
token filters: "snowball\_text", "lowercase", and "russian\_morphology". In
this case, the synonym mapping will be executing key lookups on terms that
have been stemmed and lowercase (I do not know what russian\_morphology
provides). Try moving your synonym filter before any stemming. After
lowercasing is fine, as long as your synonym map have lowercased values (or
set ignore\_case to true). In your example, foo/bar/baz have no further
stemming, so they work as is.
Cheers,
Ivan
On Thu, Mar 6, 2014 at 2:39 AM, Владимир Руденко [mailvovana@gmail.com](mailto:mailvovana@gmail.com)wrote:
> Hi.
> I have test index with settings:
> curl -XPOST '[http://localhost:9200/test\_index](http://localhost:9200/test_index)' -d '
> {
> "settings" : {
> "number\_of\_shards" : 5,
> "language":"javascript",
> "analysis": {
> "filter": {
> "snowball\_text" : {
> "type": "snowball",
> "language": "Russian"
> },
> "synonym" : {
> "type" : "synonym",
> "synonyms\_path" : "synonym.txt"
> }
> },
> "analyzer": {
> "search" : {
> "type" :"custom",
> "tokenizer": "standard",
> "filter": ["snowball\_text", "lowercase",
> "russian\_morphology", "synonym"]
> }
> }
> }
> },
> "mappings" : {
> "test\_type" : {
> "properties" : {
> "test" : {
> "type" : "string",
> "analyzer" : "search"
> },
> "description" : {
> "type" : "string",
> "analyzer" : "search"
> }
> }
> }
> }
> }'
>
> File synonym.txt:
> продажа =\> купить
> аренда =\> арендовать, сниму, снять
> foo =\> foo bar, baz
>
> English words works fine:
> curl -XGET '
> [http://localhost:9200/test\_index/\_analyze?text=foo&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=foo&analyzer=search&pretty=true)
> '
> {
> "tokens" : [ {
> "token" : "foo",
> "start\_offset" : 0,
> "end\_offset" : 3,
> "type" : "SYNONYM",
> "position" : 1
> }, {
> "token" : "baz",
> "start\_offset" : 0,
> "end\_offset" : 3,
> "type" : "SYNONYM",
> "position" : 1
> }, {
> "token" : "bar",
> "start\_offset" : 0,
> "end\_offset" : 3,
> "type" : "SYNONYM",
> "position" : 2
> } ]
> }
>
> But russian:
> curl -XGET '
> [http://localhost:9200/test\_index/\_analyze?text=продажа&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=%D0%BF%D1%80%D0%BE%D0%B4%D0%B0%D0%B6%D0%B0&analyzer=search&pretty=true)
> '
> {
> "tokens" : [ {
> "token" : "ﾀ",
> "start\_offset" : 3,
> "end\_offset" : 4,
> "type" : "",
> "position" : 1
> }, {
> "token" : "ﾾ",
> "start\_offset" : 5,
> "end\_offset" : 6,
> "type" : "",
> "position" : 2
> }, {
> "token" : "ﾴ",
> "start\_offset" : 7,
> "end\_offset" : 8,
> "type" : "",
> "position" : 3
> }, {
> "token" : "ﾰ",
> "start\_offset" : 9,
> "end\_offset" : 10,
> "type" : "",
> "position" : 4
> }, {
> "token" : "ﾶ",
> "start\_offset" : 11,
> "end\_offset" : 12,
> "type" : "",
> "position" : 5
> }, {
> "token" : "ﾰ",
> "start\_offset" : 13,
> "end\_offset" : 14,
> "type" : "",
> "position" : 6
> } ]
> }
>
> I cant't understand what i'm doing wrong?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDk5vd0kN6rNFmFwOOeTgxnrBGQo4d7GN-\_\_\_Vkj%2BRUug%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDk5vd0kN6rNFmFwOOeTgxnrBGQo4d7GN-___Vkj%2BRUug%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Ilya\_Kantor](https://discuss.elastic.co/u/Ilya_Kantor)
#### Post date: [December 22, 2014, 10:19am UTC](https://discuss.elastic.co/t/russian-words-not-work-with-synonym-token-filter/16186/3 "2014-12-22T10:19:28Z")
The topic is kind of old, but I'll answer it, just to be helpful for others
who have the similar problem.
The topicstarter used the request
curl -XGET '
[http://localhost:9200/test\_index/\_analyze?text=продажа&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=%D0%BF%D1%80%D0%BE%D0%B4%D0%B0%D0%B6%D0%B0&analyzer=search&pretty=true)
'
The mistake is that the Russian text was not urlencoded.
Elasticsearch treated it as Japanese, as clearly visible in the response.
Always urlencode Russian letters.
Cheers.
четверг, 6 марта 2014 г., 20:31:41 UTC+3 пользователь Ivan Brusic написал:
> Despite my name, I do not speak Russian. 🙂 Please excuse my ignorance of
> the Russian language while I attempt to debug.
>
> Currently, the synonym token filter is being applied after the other three
> token filters: "snowball\_text", "lowercase", and "russian\_morphology". In
> this case, the synonym mapping will be executing key lookups on terms
> that have been stemmed and lowercase (I do not know what russian\_morphology
> provides). Try moving your synonym filter before any stemming. After
> lowercasing is fine, as long as your synonym map have lowercased values (or
> set ignore\_case to true). In your example, foo/bar/baz have no further
> stemming, so they work as is.
>
> Cheers,
>
> Ivan
>
> On Thu, Mar 6, 2014 at 2:39 AM, Владимир Руденко \<[mailv...@gmail.com](mailto:mailv...@gmail.com)
> \\> wrote:
>
> > Hi.
> > I have test index with settings:
> > curl -XPOST '[http://localhost:9200/test\_index](http://localhost:9200/test_index)' -d '
> > {
> > "settings" : {
> > "number\_of\_shards" : 5,
> > "language":"javascript",
> > "analysis": {
> > "filter": {
> > "snowball\_text" : {
> > "type": "snowball",
> > "language": "Russian"
> > },
> > "synonym" : {
> > "type" : "synonym",
> > "synonyms\_path" : "synonym.txt"
> > }
> > },
> > "analyzer": {
> > "search" : {
> > "type" :"custom",
> > "tokenizer": "standard",
> > "filter": ["snowball\_text", "lowercase",
> > "russian\_morphology", "synonym"]
> > }
> > }
> > }
> > },
> > "mappings" : {
> > "test\_type" : {
> > "properties" : {
> > "test" : {
> > "type" : "string",
> > "analyzer" : "search"
> > },
> > "description" : {
> > "type" : "string",
> > "analyzer" : "search"
> > }
> > }
> > }
> > }
> > }'
> >
> > File synonym.txt:
> > продажа =\> купить
> > аренда =\> арендовать, сниму, снять
> > foo =\> foo bar, baz
> >
> > English words works fine:
> > curl -XGET '
> > [http://localhost:9200/test\_index/\_analyze?text=foo&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=foo&analyzer=search&pretty=true)
> > '
> > {
> > "tokens" : [ {
> > "token" : "foo",
> > "start\_offset" : 0,
> > "end\_offset" : 3,
> > "type" : "SYNONYM",
> > "position" : 1
> > }, {
> > "token" : "baz",
> > "start\_offset" : 0,
> > "end\_offset" : 3,
> > "type" : "SYNONYM",
> > "position" : 1
> > }, {
> > "token" : "bar",
> > "start\_offset" : 0,
> > "end\_offset" : 3,
> > "type" : "SYNONYM",
> > "position" : 2
> > } ]
> > }
> >
> > But russian:
> > curl -XGET '
> > [http://localhost:9200/test\_index/\_analyze?text=продажа&analyzer=search&pretty=true](http://localhost:9200/test_index/_analyze?text=%D0%BF%D1%80%D0%BE%D0%B4%D0%B0%D0%B6%D0%B0&analyzer=search&pretty=true)
> > '
> > {
> > "tokens" : [ {
> > "token" : "ﾀ",
> > "start\_offset" : 3,
> > "end\_offset" : 4,
> > "type" : "",
> > "position" : 1
> > }, {
> > "token" : "ﾾ",
> > "start\_offset" : 5,
> > "end\_offset" : 6,
> > "type" : "",
> > "position" : 2
> > }, {
> > "token" : "ﾴ",
> > "start\_offset" : 7,
> > "end\_offset" : 8,
> > "type" : "",
> > "position" : 3
> > }, {
> > "token" : "ﾰ",
> > "start\_offset" : 9,
> > "end\_offset" : 10,
> > "type" : "",
> > "position" : 4
> > }, {
> > "token" : "ﾶ",
> > "start\_offset" : 11,
> > "end\_offset" : 12,
> > "type" : "",
> > "position" : 5
> > }, {
> > "token" : "ﾰ",
> > "start\_offset" : 13,
> > "end\_offset" : 14,
> > "type" : "",
> > "position" : 6
> > } ]
> > }
> >
> > I cant't understand what i'm doing wrong?
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/481aacc3-d892-43e3-9024-65d84dcffe56%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8fa48048-8fec-414a-b3c3-4667c38b2b93%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8fa48048-8fec-414a-b3c3-4667c38b2b93%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:42am UTC](https://discuss.elastic.co/t/russian-words-not-work-with-synonym-token-filter/16186/4 "2017-07-06T00:42:22Z")