How to use my customer lucene analyzer(tokenizer)?

fancyerii · August 5, 2014, 6:58am

I want to use my own Chinese analyzer and I can write lucene analyzer class
myself. How can I integrate it to elasticsearch?
I googled and
found http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-analyzers.html.
But it only combine existing tokenizers and filters. I can use tokenizer
writing in java by myself.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tihomir_Lichev1 · August 5, 2014, 7:30am

I think you should write your own plugin with your analyzer implementation,
so you can register your analyzer into the ES and use it later on.
You could have a look and see how it done maybe in this
plugin: GitHub - elastic/elasticsearch-analysis-kuromoji: Japanese (kuromoji) Analysis Plugin

05 август 2014, вторник, 09:58:16 UTC+3, fanc...@gmail.com написа:

I want to use my own Chinese analyzer and I can write lucene analyzer
class myself. How can I integrate it to elasticsearch?
I googled and found
Elasticsearch Platform — Find real-time answers at scale | Elastic.
But it only combine existing tokenizers and filters. I can use tokenizer
writing in java by myself.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fe02dc31-991c-49f9-9820-63ba96a37d33%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

johtani · August 5, 2014, 7:34am

Hi,

I think this plugin will be helpful for you.

2014/08/05 15:58 fancyerii@gmail.com:

I want to use my own Chinese analyzer and I can write lucene analyzer
class myself. How can I integrate it to elasticsearch?
I googled and found
Elasticsearch Platform — Find real-time answers at scale | Elastic.
But it only combine existing tokenizers and filters. I can use tokenizer
writing in java by myself.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPW8A5xqnsWa76TJuSAWwxPyrUA8F9cJ6tyTs2ZJxXFpb4vEMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

art · August 22, 2014, 12:18am

I have the same question about using an analyzer I have written as a
plug-in for Elasticsearch 1.3.

github.com

elastic/elasticsearch-analysis-kuromoji/blob/es-1.3/README.md

Japanese (kuromoji) Analysis for Elasticsearch
==================================

The Japanese (kuromoji) Analysis plugin integrates Lucene kuromoji analysis module into elasticsearch.

In order to install the plugin, simply run: `bin/plugin -install elasticsearch/elasticsearch-analysis-kuromoji/2.3.0`.

* For master elasticsearch versions, look at [master branch](https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/tree/master).
* For 1.3.x elasticsearch versions, look at [es-1.3 branch](https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/tree/es-1.3).
* For 1.2.x elasticsearch versions, look at [es-1.2 branch](https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/tree/es-1.2).
* For 1.1.x elasticsearch versions, look at [es-1.1 branch](https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/tree/es-1.1).
* For 1.0.x elasticsearch versions, look at [es-1.0 branch](https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/tree/es-1.0).
* For 0.90.x elasticsearch versions, look at [es-0.90 branch](https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/tree/es-0.90).


|  Kuromoji Analysis Plugin   |    elasticsearch    | Release date |
|-----------------------------|---------------------|:------------:|
| 2.3.1-SNAPSHOT              | 1.3.0 -> 1.3        |  XXXX-XX-XX  |
| 2.3.0                       | 1.3.0 -> 1.3        |  2014-07-23  |

This file has been truncated. show original

demonstrates only how to use the tokenizers in combination with the
built-in CustomAnalyzer. They do not show how to use the kuromoji analyzer
itself.

When I try to specify my analyzer for a field, I get errors like this:

MapperParsingException[Analyzer [special_analyzer] not found for field
[foo]];

Can you show an example of how to specify the kuromoji analyzer for a
field? I should then be able to adapt it for use with my plugin analyzer.

Thanks in advance,
Art

On Tuesday, August 5, 2014 12:34:42 AM UTC-7, Jun Ohtani wrote:

Hi,

I think this plugin will be helpful for you.

GitHub - elastic/elasticsearch-analysis-kuromoji: Japanese (kuromoji) Analysis Plugin
2014/08/05 15:58 <fanc...@gmail.com <javascript:>>:

I want to use my own Chinese analyzer and I can write lucene analyzer
class myself. How can I integrate it to elasticsearch?
I googled and found
Elasticsearch Platform — Find real-time answers at scale | Elastic.
But it only combine existing tokenizers and filters. I can use tokenizer
writing in java by myself.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/da795847-3ea2-4afb-9a7b-aefdd6f111a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

johtani · August 22, 2014, 6:23am

Hi Art,

I wrote an example specifying the kuromoji analyzer("kuromoji") and custom
analyzer("my_analyzer") for a field.

curl -XPUT "http://localhost:9200/kuromoji-sample" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "kuromoji_tokenizer",
"filter": [
"kuromoji_baseform"
]
}
}
}
}
},
"mappings": {
"sample": {
"properties": {
"title": {
"type": "string",
"analyzer": "my_analyzer"
},
"body" : {
"type": "string",
"analyzer": "kuromoji"
}
}
}
}
}'

I hope that it will be helpful for you.

2014-08-22 9:18 GMT+09:00 art@safeshepherd.com:

I have the same question about using an analyzer I have written as a
plug-in for Elasticsearch 1.3.

https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/blob/es-1.3/README.md
demonstrates only how to use the tokenizers in combination with the
built-in CustomAnalyzer. They do not show how to use the kuromoji analyzer
itself.

When I try to specify my analyzer for a field, I get errors like this:

MapperParsingException[Analyzer [special_analyzer] not found for field
[foo]];

Can you show an example of how to specify the kuromoji analyzer for a
field? I should then be able to adapt it for use with my plugin analyzer.

Thanks in advance,
Art

On Tuesday, August 5, 2014 12:34:42 AM UTC-7, Jun Ohtani wrote:

Hi,

I think this plugin will be helpful for you.

GitHub - elastic/elasticsearch-analysis-kuromoji: Japanese (kuromoji) Analysis Plugin
2014/08/05 15:58 fanc...@gmail.com:

I want to use my own Chinese analyzer and I can write lucene analyzer
class myself. How can I integrate it to elasticsearch?
I googled and found Elasticsearch Platform — Find real-time answers at scale | Elastic
elasticsearch/guide/current/custom-analyzers.html. But it only combine
existing tokenizers and filters. I can use tokenizer writing in java by
myself.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/da795847-3ea2-4afb-9a7b-aefdd6f111a0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/da795847-3ea2-4afb-9a7b-aefdd6f111a0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Jun Ohtani
blog : http://blog.johtani.info

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPW8A5x9xr_4OoHeBjCO%2BYYJHN3-O5pSk2fB7-v0rETKPMRHkg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

art · August 26, 2014, 5:08pm

Thanks Jun, that was helpful. It helped me to realize I had not fully
connected my analyzer plugin.

On Thursday, August 21, 2014 11:23:47 PM UTC-7, Jun Ohtani wrote:

Hi Art,

I wrote an example specifying the kuromoji analyzer("kuromoji") and custom
analyzer("my_analyzer") for a field.

curl -XPUT "http://localhost:9200/kuromoji-sample" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "kuromoji_tokenizer",
"filter": [
"kuromoji_baseform"
]
}
}
}
}
},
"mappings": {
"sample": {
"properties": {
"title": {
"type": "string",
"analyzer": "my_analyzer"
},
"body" : {
"type": "string",
"analyzer": "kuromoji"
}
}
}
}
}'

I hope that it will be helpful for you.

2014-08-22 9:18 GMT+09:00 <a...@safeshepherd.com <javascript:>>:

I have the same question about using an analyzer I have written as a
plug-in for Elasticsearch 1.3.

https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/blob/es-1.3/README.md
demonstrates only how to use the tokenizers in combination with the
built-in CustomAnalyzer. They do not show how to use the kuromoji analyzer
itself.

When I try to specify my analyzer for a field, I get errors like this:

MapperParsingException[Analyzer [special_analyzer] not found for field
[foo]];

Can you show an example of how to specify the kuromoji analyzer for a
field? I should then be able to adapt it for use with my plugin analyzer.

Thanks in advance,
Art

On Tuesday, August 5, 2014 12:34:42 AM UTC-7, Jun Ohtani wrote:

Hi,

I think this plugin will be helpful for you.

GitHub - elastic/elasticsearch-analysis-kuromoji: Japanese (kuromoji) Analysis Plugin
2014/08/05 15:58 fanc...@gmail.com:

I want to use my own Chinese analyzer and I can write lucene analyzer
class myself. How can I integrate it to elasticsearch?
I googled and found Elasticsearch Platform — Find real-time answers at scale | Elastic
elasticsearch/guide/current/custom-analyzers.html. But it only combine
existing tokenizers and filters. I can use tokenizer writing in java by
myself.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c3fe52cd-8cb5-4c53-b0fe-87183deb45bf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/da795847-3ea2-4afb-9a7b-aefdd6f111a0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/da795847-3ea2-4afb-9a7b-aefdd6f111a0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Jun Ohtani
blog : http://blog.johtani.info

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a792d08d-534f-4619-bfcb-0f01262b6c51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Smart chinese analyzer Elasticsearch	5	394	July 6, 2017
Arabic Tokenizer Elasticsearch	4	2618	July 6, 2017
Utilizing other lucene analyzers (eg stanford lemmatizer) Elasticsearch	2	516	July 6, 2017
Elasticsearch and plugins Elasticsearch	2	363	July 6, 2017
Custom tokenizer in .NET language, push already tokenized text into ES Elasticsearch	2	354	July 6, 2017

How to use my customer lucene analyzer(tokenizer)?

--

--

Related topics