Adding analyzers during runtime

vpunski · October 11, 2011, 1:33pm

The message was originally published on nabble.com, but not appeared
here.
Sorry for potential double posting.

I have an index of 65 millions records working on production system.
In a new version of the application, I'd like to make some changes in
query logic, that needs some mapping updates.
To be more precise, I'd like to to change standard analyzer of
particular field to analyzer based on path_hierarchy tokenizer.
I understand that the change needs reindexing of the field, or even
field duplication, and I'm ok with that. This is not a problem. The
real problem is that I can't figure out how to configure new analyzer
on existing index, and update mapping by adding new field analyzed by
a new analyzer.
Is it possible? There is no such option in index _settings API.
From my point of view, updating mapping by adding new fields,
including merging mechanism is a killer feature. By adding the
possibility to use new analyzers for new fields, existing "dynamic
schema" nature gets additional feature - "dynamic capabilities".
Am I missing something in theory?
I didn't find anything searching over the mail list related.

Thanks

ppearcy · October 12, 2011, 5:49pm

Hey Vadim,
It is possible to add analyzers at runtime to new indexes, but I
don't think it is currently possible to dynamically add analyzers to
existing indices. I tried with the update settings API and couldn't
get it working, but I really don't know if my request was correct.
This would be a nice feature though.

If you're curious, this gist shows adding a new index with dynamic
analyzer and what I attempted to do to add another analyzer to that
index:

gist.github.com

https://gist.github.com/ppearcy/1281923

gistfile1.txt

# Create new index with dynamic analyzer - works
curl -XPOST localhost:9200/es-syntest -d '{
    "settings" : {
        "number_of_shards" : 1,
	"analysis" : {"analyzer":{"synonym_test_analyzer":{"type":"custom","tokenizer":"standard","filter":["synonym_test"]}},"filter":{"synonym_test":{"type":"synonym","synonyms_path":"analysis/synonym_test.txt","ignore_case":true,"expand":false}}}
    },
    "mappings" : {
        "es-syntest" : {
            "properties" : {
                "attachmentcontent" : { "type" : "string", "omit_norms" : true, "include_in_all" : false, "analyzer" :  "synonym_test_analyzer", "term_vector" : "with_positions_offsets"}

This file has been truncated. show original

The only current workaround I know of is to add the analyzer to the
config file, restart the cluster, and then add mappings for that new
analyzer.

Best Regards,
Paul

On Oct 11, 7:33 am, vadim vpun...@gmail.com wrote:

The message was originally published on nabble.com, but not appeared
here.
Sorry for potential double posting.

I have an index of 65 millions records working on production system.
In a new version of the application, I'd like to make some changes in
query logic, that needs some mapping updates.
To be more precise, I'd like to to change standard analyzer of
particular field to analyzer based on path_hierarchy tokenizer.
I understand that the change needs reindexing of the field, or even
field duplication, and I'm ok with that. This is not a problem. The
real problem is that I can't figure out how to configure new analyzer
on existing index, and update mapping byaddingnew field analyzed by
a new analyzer.
Is it possible? There is no such option in index _settings API.
From my point of view, updating mapping byaddingnew fields,
including merging mechanism is a killer feature. Byaddingthe
possibility to use newanalyzersfor new fields, existing "dynamic
schema" nature gets additional feature - "dynamic capabilities".
Am I missing something in theory?
I didn't find anything searching over the mail list related.

Thanks

kimchy · October 12, 2011, 9:25pm

You can add new analyzers to an index using the update settings API (the
update settings API should have only the config for the new analyzer, as
they are merged). But, the index has to be closed to do that (i.e. its not
a runtime/realtime update setting).

On Wed, Oct 12, 2011 at 7:49 PM, ppearcy ppearcy@gmail.com wrote:

Hey Vadim,
It is possible to add analyzers at runtime to new indexes, but I
don't think it is currently possible to dynamically add analyzers to
existing indices. I tried with the update settings API and couldn't
get it working, but I really don't know if my request was correct.
This would be a nice feature though.

If you're curious, this gist shows adding a new index with dynamic
analyzer and what I attempted to do to add another analyzer to that
index:
gist:1281923 · GitHub

The only current workaround I know of is to add the analyzer to the
config file, restart the cluster, and then add mappings for that new
analyzer.

Best Regards,
Paul

On Oct 11, 7:33 am, vadim vpun...@gmail.com wrote:

The message was originally published on nabble.com, but not appeared
here.
Sorry for potential double posting.

I have an index of 65 millions records working on production system.
In a new version of the application, I'd like to make some changes in
query logic, that needs some mapping updates.
To be more precise, I'd like to to change standard analyzer of
particular field to analyzer based on path_hierarchy tokenizer.
I understand that the change needs reindexing of the field, or even
field duplication, and I'm ok with that. This is not a problem. The
real problem is that I can't figure out how to configure new analyzer
on existing index, and update mapping byaddingnew field analyzed by
a new analyzer.
Is it possible? There is no such option in index _settings API.
From my point of view, updating mapping byaddingnew fields,
including merging mechanism is a killer feature. Byaddingthe
possibility to use newanalyzersfor new fields, existing "dynamic
schema" nature gets additional feature - "dynamic capabilities".
Am I missing something in theory?
I didn't find anything searching over the mail list related.

Thanks