Merge_policy not in settings


(ruflin-2) #1

Hi

If I update my merge_policy settings and then later try to retrieve the
settings with the code below, merge_policy does not appear in the result
array. This is the case since ES 0.17.0.

curl -XGET 'http://localhost:9200/test/_settings'


(Shay Banon) #2

Can you gist an example of what you do exactly? You can't change the merge
policy on an open index, only on a closed one.

On Fri, Jul 29, 2011 at 6:24 PM, Nicolas Ruflin ruflin@gmail.com wrote:

Hi

If I update my merge_policy settings and then later try to retrieve the
settings with the code below, merge_policy does not appear in the result
array. This is the case since ES 0.17.0.

curl -XGET 'http://localhost:9200/test/_settings'


(Nicolas Ruflin) #3

This probably explains the problem. I will try it with a closed index tomorrow. I tried to do it on an open index. Did this work on 0.16.* or did it only "seem" to work?

useKit.com
Xodoa.com

On 29.07.2011, at 20:36, Shay Banon wrote:

Can you gist an example of what you do exactly? You can't change the merge policy on an open index, only on a closed one.

On Fri, Jul 29, 2011 at 6:24 PM, Nicolas Ruflin ruflin@gmail.com wrote:
Hi

If I update my merge_policy settings and then later try to retrieve the settings with the code below, merge_policy does not appear in the result array. This is the case since ES 0.17.0.

curl -XGET 'http://localhost:9200/test/_settings'


(Shay Banon) #4

Seem to work :slight_smile:

On Fri, Jul 29, 2011 at 11:36 PM, Nicolas Ruflin mail@ruflin.com wrote:

This probably explains the problem. I will try it with a closed index
tomorrow. I tried to do it on an open index. Did this work on 0.16.* or did
it only "seem" to work?

useKit.com
Xodoa.com http://xodoa.com/
Ruflin.com http://ruflin.com

On 29.07.2011, at 20:36, Shay Banon wrote:

Can you gist an example of what you do exactly? You can't change the merge
policy on an open index, only on a closed one.

On Fri, Jul 29, 2011 at 6:24 PM, Nicolas Ruflin ruflin@gmail.com wrote:

Hi

If I update my merge_policy settings and then later try to retrieve the
settings with the code below, merge_policy does not appear in the result
array. This is the case since ES 0.17.0.

curl -XGET 'http://localhost:9200/test/_settings'


(ruflin-2) #5

Ok, closing the index solved the problem. This also explains the outcome of
my performance tests under 0.16.* :wink:

For which other settings the index has to be closed first?

Thanks for your help.


(Shay Banon) #6

The realtime settings are listed in the update settings API:
http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html.
Also, update settings will log settings that have not been applied.

On Sat, Jul 30, 2011 at 1:22 PM, ruflin ruflin@gmail.com wrote:

Ok, closing the index solved the problem. This also explains the outcome of
my performance tests under 0.16.* :wink:

For which other settings the index has to be closed first?

Thanks for your help.


(ruflin-2) #7

Ok, now it makes more and more sense. Probably I should check the docu more
frequently for changes. I will add some notes about this to Elastica.

What do you exactly mean by log settings that have not been applied? Can it
happen that the settings are changed and they are not already applied at the
next request? Would refresh help?

Btw: It is a little bit confusing that merge policy is also listed in the
table but can't be changed (I know, it's on an other page). Perhaps a small
note would help. And their is a small typo in policy in the table.

One more question: The three different merge policy types, do they exist as
templates and is it possible to change between these templates, or is just a
definition? How does a switch happen?


(Shay Banon) #8

On Sat, Jul 30, 2011 at 10:05 PM, ruflin ruflin@gmail.com wrote:

Ok, now it makes more and more sense. Probably I should check the docu more
frequently for changes. I will add some notes about this to Elastica.

What do you exactly mean by log settings that have not been applied? Can it
happen that the settings are changed and they are not already applied at the
next request? Would refresh help?

I mean settings that are not supported in being changed on an open index.

Btw: It is a little bit confusing that merge policy is also listed in the
table but can't be changed (I know, it's on an other page). Perhaps a small
note would help. And their is a small typo in policy in the table.

Yea, you are right. I added a better description.

One more question: The three different merge policy types, do they exist as
templates and is it possible to change between these templates, or is just a
definition? How does a switch happen?

Not sure I understand the question...


(ruflin-2) #9

Ok, I try the last question again :wink:

Here you have listed three different types of merge policies (tiered,
log_by_size, log_doc):
http://www.elasticsearch.org/guide/reference/index-modules/merge.html

Are these predefined merge policy templates and is it somehow possible to
switch between these templates through the API? Or is it necessary to change
all the settings one by one to get to the other merge policy type?


(Shay Banon) #10

The default merge policy in 0.17 is the tiered one (a new merge policy
introduced in Lucene 3.3, which 0.17 comes with). If you want to change the
merge policy type, you need to change it on a closed index, by setting the
relevant setting for it, for example, set index.merge.policy.type to
log_by_size. On an open index, you can change all the settings associated
with the current merge policy used.

On Mon, Aug 1, 2011 at 1:36 AM, ruflin ruflin@gmail.com wrote:

Ok, I try the last question again :wink:

Here you have listed three different types of merge policies (tiered,
log_by_size, log_doc):
http://www.elasticsearch.org/guide/reference/index-modules/merge.html

Are these predefined merge policy templates and is it somehow possible to
switch between these templates through the API? Or is it necessary to change
all the settings one by one to get to the other merge policy type?


(ruflin-2) #11

This is getting more and more interesting. This means, I can change
merge_policy on an open index, if the policy type is log_bytes_size?

I tried to change the policy type to log_bytes_size but this lead to an
endless loop in elasticsearch (even though it was returned successfully). I
figured out the following:
In the documentation it is named log_bytes_size. But the java class name is
LogByteSize (singular). I also tried to set the policy value with
lob_byte_size or LogByteSize, both didn't work.

Elasticsearch throws out (in an endless loop) the following message, until
the index is deleted:

[2011-08-01 12:28:09,044][WARN ][cluster.action.shard ] [Husk] sending
failed shard for [test][2], node[MxR7mlleRNaTbruLG2A_-w], [P],
s[INITIALIZING], reason [Failed to create shard, message
[CreationException[Guice creation errors:

  1. An exception was caught and reported. Message: Failed to load class
    setting [index.merge.policy.type] with value [LogByteSize]
    at
    org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:142)

  2. No implementation for
    org.elasticsearch.index.merge.policy.MergePolicyProvider was bound.
    at
    org.elasticsearch.index.merge.policy.MergePolicyModule.configure(MergePolicyModule.java:37)

2 errors]; nested: NoClassSettingsException[Failed to load class setting
[index.merge.policy.type] with value [LogByteSize]]; nested:
ClassNotFoundException[LogByteSize]; ]]
[2011-08-01 12:28:09,044][WARN ][cluster.action.shard ] [Husk] received
shard failed for [test][2], node[MxR7mlleRNaTbruLG2A_-w], [P],
s[INITIALIZING], reason [Failed to create shard, message
[CreationException[Guice creation errors:

  1. An exception was caught and reported. Message: Failed to load class
    setting [index.merge.policy.type] with value [LogByteSize]
    at
    org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:142)

  2. No implementation for
    org.elasticsearch.index.merge.policy.MergePolicyProvider was bound.
    at
    org.elasticsearch.index.merge.policy.MergePolicyModule.configure(MergePolicyModule.java:37)

2 errors]; nested: NoClassSettingsException[Failed to load class setting
[index.merge.policy.type] with value [LogByteSize]]; nested:
ClassNotFoundException[LogByteSize]; ]]
Aug 1, 2011 12:28:09 PM org.elasticsearch.common.inject.MessageProcessor
visit
INFO: An exception was caught and reported. Message:
java.lang.ClassNotFoundException: LogByteSize
org.elasticsearch.common.settings.NoClassSettingsException: Failed to load
class setting [index.merge.policy.type] with value [LogByteSize]
at
org.elasticsearch.common.settings.ImmutableSettings.getAsClass(ImmutableSettings.java:224)
at
org.elasticsearch.index.merge.policy.MergePolicyModule.configure(MergePolicyModule.java:37)
at
org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:59)
at
org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:210)
at
org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:91)
at
org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:142)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:103)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:138)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:65)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:288)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:526)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:495)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:178)
at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:254)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException: LogByteSize
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.elasticsearch.common.settings.ImmutableSettings.getAsClass(ImmutableSettings.java:214)
... 16 more


(Shay Banon) #12

First, there is a bug in the docs regarding the setting name, it should be
log_byte_size. I fixed it, but there is still a bug when setting it using
the mentioned descriptive value, opened an issue nad pushed a fix:
https://github.com/elasticsearch/elasticsearch/issues/1190.

Regarding changing settings. On an open index, you can only change the
settings of a specific type chosen. You can't change the type itself. If the
current merge policy of hte index is tiered, you can change tiered level
settings, and they will be applied, if its log_byte_size, then you can
change log_byte_size relevant settings and they will be applied.

On Mon, Aug 1, 2011 at 1:33 PM, ruflin ruflin@gmail.com wrote:

This is getting more and more interesting. This means, I can change
merge_policy on an open index, if the policy type is log_bytes_size?

I tried to change the policy type to log_bytes_size but this lead to an
endless loop in elasticsearch (even though it was returned successfully). I
figured out the following:
In the documentation it is named log_bytes_size. But the java class name is
LogByteSize (singular). I also tried to set the policy value with
lob_byte_size or LogByteSize, both didn't work.

Elasticsearch throws out (in an endless loop) the following message, until
the index is deleted:

[2011-08-01 12:28:09,044][WARN ][cluster.action.shard ] [Husk] sending
failed shard for [test][2], node[MxR7mlleRNaTbruLG2A_-w], [P],
s[INITIALIZING], reason [Failed to create shard, message
[CreationException[Guice creation errors:

  1. An exception was caught and reported. Message: Failed to load class
    setting [index.merge.policy.type] with value [LogByteSize]
    at
    org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:142)

  2. No implementation for
    org.elasticsearch.index.merge.policy.MergePolicyProvider was bound.
    at
    org.elasticsearch.index.merge.policy.MergePolicyModule.configure(MergePolicyModule.java:37)

2 errors]; nested: NoClassSettingsException[Failed to load class setting
[index.merge.policy.type] with value [LogByteSize]]; nested:
ClassNotFoundException[LogByteSize]; ]]
[2011-08-01 12:28:09,044][WARN ][cluster.action.shard ] [Husk] received
shard failed for [test][2], node[MxR7mlleRNaTbruLG2A_-w], [P],
s[INITIALIZING], reason [Failed to create shard, message
[CreationException[Guice creation errors:

  1. An exception was caught and reported. Message: Failed to load class
    setting [index.merge.policy.type] with value [LogByteSize]
    at
    org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:142)

  2. No implementation for
    org.elasticsearch.index.merge.policy.MergePolicyProvider was bound.
    at
    org.elasticsearch.index.merge.policy.MergePolicyModule.configure(MergePolicyModule.java:37)

2 errors]; nested: NoClassSettingsException[Failed to load class setting
[index.merge.policy.type] with value [LogByteSize]]; nested:
ClassNotFoundException[LogByteSize]; ]]
Aug 1, 2011 12:28:09 PM org.elasticsearch.common.inject.MessageProcessor
visit
INFO: An exception was caught and reported. Message:
java.lang.ClassNotFoundException: LogByteSize
org.elasticsearch.common.settings.NoClassSettingsException: Failed to load
class setting [index.merge.policy.type] with value [LogByteSize]
at
org.elasticsearch.common.settings.ImmutableSettings.getAsClass(ImmutableSettings.java:224)
at
org.elasticsearch.index.merge.policy.MergePolicyModule.configure(MergePolicyModule.java:37)
at
org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:59)
at
org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:210)
at
org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:91)
at
org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:142)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:103)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:138)
at
org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:65)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:288)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:526)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:495)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:178)
at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:254)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException: LogByteSize
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.elasticsearch.common.settings.ImmutableSettings.getAsClass(ImmutableSettings.java:214)
... 16 more


(ruflin-2) #13

Thx for the fast fix. I will try everything as soon as I get some time.
Probably then already 0.17.3 is released :wink:


(ruflin-2) #14

I just tried out setting a type and everything with ES 0.17.3. Everything
seems to work. But now I can set any setting I want (also not existing once)
and it will return true. So I don't really know, if the type was changed or
not. I would expect that I get an error returned, if I try to set a not
existing setting. Or is this as expected?


(ruflin-2) #15

I just tried it with 0.17.6 again, and I'm able to set any not existing
settings and elasticsearch returns ok (and also stores the not existing
setting). Can you give me some more details here what should be expected?


(Shay Banon) #16

Can you gist a curl sample of what you do?

On Mon, Aug 15, 2011 at 9:04 AM, ruflin ruflin@gmail.com wrote:

I just tried it with 0.17.6 again, and I'm able to set any not existing
settings and elasticsearch returns ok (and also stores the not existing
setting). Can you give me some more details here what should be expected?


(ruflin-2) #17

Here it is:

I'm suprised by the last line of the output (hello_world settings). Also as
I unserstood you before, it should not be possible to
set expunge_deletes_allowed on an open index. But it also appears in the
settings and returns ok.


(system) #18