This Mornings Snapshot


(Kenneth Loafman) #1

I upgraded this morning to the new snapshot and it's continuously
dumping while restoring:

[06:52:04,019][WARN ][indices.cluster ] [Brown, Abe]
[test-index][1] failed to create shard
org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
at
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
at
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
at
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
at
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
at
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
at
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
at
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
at
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
at
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
at
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
at java.lang.Class.getDeclaredConstructors(Class.java:1853)
at
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
at
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
at
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
at
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
... 26 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 36 more
[06:52:04,019][WARN ][cluster.action.shard ] [Brown, Abe] sending
failed shard for [test-index][1],
node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P], s[INITIALIZING], reason
[Failed to create shard, message
[ComputationException[java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler]; nested:
NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];
nested:
ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];
]]

...Ken


(Shay Banon) #2

Can you run a "gradlew clean release" (the clean is important)?

-shay.banon

On Sun, Aug 22, 2010 at 3:20 PM, Kenneth Loafman
kenneth.loafman@gmail.comwrote:

I upgraded this morning to the new snapshot and it's continuously
dumping while restoring:

[06:52:04,019][WARN ][indices.cluster ] [Brown, Abe]
[test-index][1] failed to create shard
org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
at

org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
at

org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
at

org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
at

org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
at

org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at

org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
at

org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
at

org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
at
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
at

org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
at

org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
at

org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
at

org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
at

org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
at

org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
at

org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
at java.lang.Class.getDeclaredConstructors(Class.java:1853)
at

org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
at

org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
at

org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
at

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
at

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
at

org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
... 26 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 36 more
[06:52:04,019][WARN ][cluster.action.shard ] [Brown, Abe] sending
failed shard for [test-index][1],
node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P], s[INITIALIZING], reason
[Failed to create shard, message
[ComputationException[java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler]; nested:

NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];
nested:

ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];
]]

...Ken


(Kenneth Loafman) #3

Same thing, but maybe a bit longer traceback...

[10:09:38,606][WARN ][indices.cluster ] [USAgent] [google][0] failed to create shard
org.elasticsearch.common.inject.internal.ComputationException: java.lang.NoClassDefFoundError: org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
at org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
at org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
at org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
at org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
at org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
at org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
at org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
at org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
at org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
at org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
at org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
at org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
at org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
at java.lang.Class.getDeclaredConstructors(Class.java:1853)
at org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
at org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
at org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
at org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
at org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
... 26 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 36 more

...Ken

Shay Banon wrote:

Can you run a "gradlew clean release" (the clean is important)?

-shay.banon

On Sun, Aug 22, 2010 at 3:20 PM, Kenneth Loafman
<kenneth.loafman@gmail.com mailto:kenneth.loafman@gmail.com> wrote:

I upgraded this morning to the new snapshot and it's continuously
dumping while restoring:

[06:52:04,019][WARN ][indices.cluster          ] [Brown, Abe]
[test-index][1] failed to create shard
org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
   at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
   at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
   at
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
   at
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
   at
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
   at
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
   at
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
   at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
   at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
   at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
   at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
   at
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
   at
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
   at
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
   at
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
   at
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
   at
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
   at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
   at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
   at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
   at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
   at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
   at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
   at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
   at java.lang.Class.getDeclaredConstructors0(Native Method)
   at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
   at java.lang.Class.getDeclaredConstructors(Class.java:1853)
   at
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
   at
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
   at
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
   at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
   at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
   at
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
   at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
   ... 26 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
   at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
   ... 36 more
[06:52:04,019][WARN ][cluster.action.shard     ] [Brown, Abe] sending
failed shard for [test-index][1],
node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P], s[INITIALIZING], reason
[Failed to create shard, message
[ComputationException[java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler]; nested:
NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];
nested:
ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];
]]

...Ken

(Shay Banon) #4

Did a fresh clone of master, ran "gradle release", opened the distribution,
started elasticsearch, and created an index on it. All seems to work...

-shay.banon

On Sun, Aug 22, 2010 at 6:12 PM, Kenneth Loafman
kenneth.loafman@gmail.comwrote:

Same thing, but maybe a bit longer traceback...

[10:09:38,606][WARN ][indices.cluster ] [USAgent] [google][0]
failed to create shard
org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
at
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
at
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
at
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
at
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
at
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
at
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
at
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
at
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
at
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
at
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
at
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
at java.lang.Class.getDeclaredConstructors0(Native Method)
at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
at java.lang.Class.getDeclaredConstructors(Class.java:1853)
at
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
at
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
at
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
at
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
... 26 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 36 more

...Ken

Shay Banon wrote:

Can you run a "gradlew clean release" (the clean is important)?

-shay.banon

On Sun, Aug 22, 2010 at 3:20 PM, Kenneth Loafman
<kenneth.loafman@gmail.com mailto:kenneth.loafman@gmail.com> wrote:

I upgraded this morning to the new snapshot and it's continuously
dumping while restoring:

[06:52:04,019][WARN ][indices.cluster          ] [Brown, Abe]
[test-index][1] failed to create shard
org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
   at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)

   at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)

   at

org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)

   at

org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)

   at

org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)

   at

org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)

   at

org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)

   at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)

   at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)

   at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)

   at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)

   at

org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)

   at

org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)

   at

org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)

   at

org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)

   at

org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)

   at

org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)

   at

org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)

   at

org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)

   at

org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)

   at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)

   at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)

   at

org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)

   at

org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)

   at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

   at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

   at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
   at java.lang.Class.getDeclaredConstructors0(Native Method)
   at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
   at java.lang.Class.getDeclaredConstructors(Class.java:1853)
   at

org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)

   at

org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)

   at

org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)

   at

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)

   at

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)

   at

org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)

   at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)

   ... 26 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
   at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
   ... 36 more
[06:52:04,019][WARN ][cluster.action.shard     ] [Brown, Abe] sending
failed shard for [test-index][1],
node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P], s[INITIALIZING],

reason

[Failed to create shard, message
[ComputationException[java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];

nested:

NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];

nested:

ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];

]]

...Ken

(Kenneth Loafman) #5

For some reason when I do the git clone, I don't end up with a
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
anywhere.

Here's the steps I took to re-clone Elastisearch:
git clone git://github.com/elasticsearch/elasticsearch.git
./gradlew clean release
unzip build/distributions/elasticsearch-0.9.1-SNAPSHOT.zip
cd elasticsearch-0.9.1-SNAPSHOT/
for N in 0 1 2; do
cp -rp bin lib /opt/es-node$N/
done

Then I restarted via:
for N in 0 1 2; do
/opt/es-node$N/bin/service/elasticsearch restart
done

Do you see anything obviously wrong?

...Ken

Shay Banon wrote:

Did a fresh clone of master, ran "gradle release", opened the
distribution, started elasticsearch, and created an index on it. All
seems to work...

-shay.banon

On Sun, Aug 22, 2010 at 6:12 PM, Kenneth Loafman
<kenneth.loafman@gmail.com mailto:kenneth.loafman@gmail.com> wrote:

Same thing, but maybe a bit longer traceback...

> [10:09:38,606][WARN ][indices.cluster          ] [USAgent]
[google][0] failed to create shard
> org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>         at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>         at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>         at
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>         at
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
>         at
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
>         at
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>         at
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
>         at
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
>         at
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
>         at
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
>         at
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
>         at
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
>         at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
>         at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
>         at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
>         at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
>         at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
>         at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
>         at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
>         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>         at java.lang.Class.getDeclaredConstructors0(Native Method)
>         at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
>         at java.lang.Class.getDeclaredConstructors(Class.java:1853)
>         at
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
>         at
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
>         at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
>         ... 26 more
> Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>         at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>         ... 36 more

...Ken

Shay Banon wrote:
> Can you run a "gradlew clean release" (the clean is important)?
>
> -shay.banon
>
> On Sun, Aug 22, 2010 at 3:20 PM, Kenneth Loafman
> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
<mailto:kenneth.loafman@gmail.com
<mailto:kenneth.loafman@gmail.com>>> wrote:
>
>     I upgraded this morning to the new snapshot and it's continuously
>     dumping while restoring:
>
>     [06:52:04,019][WARN ][indices.cluster          ] [Brown, Abe]
>     [test-index][1] failed to create shard
>     org.elasticsearch.common.inject.internal.ComputationException:
>     java.lang.NoClassDefFoundError:
>     org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>        at
>    
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>        at
>    
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>        at
>    
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>        at
>    
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
>        at
>    
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
>        at
>    
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
>        at
>    
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
>        at
>    
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
>        at
>    
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
>        at
>    
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
>        at
>    
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
>        at
>    
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
>        at
>    
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
>        at
>    
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
>        at
>    
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
>    
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
>     Caused by: java.lang.NoClassDefFoundError:
>     org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>        at java.lang.Class.getDeclaredConstructors0(Native Method)
>        at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
>        at java.lang.Class.getDeclaredConstructors(Class.java:1853)
>        at
>    
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
>        at
>    
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
>        at
>    
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
>        ... 26 more
>     Caused by: java.lang.ClassNotFoundException:
>     org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>        at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>        ... 36 more
>     [06:52:04,019][WARN ][cluster.action.shard     ] [Brown, Abe]
sending
>     failed shard for [test-index][1],
>     node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P],
s[INITIALIZING], reason
>     [Failed to create shard, message
>     [ComputationException[java.lang.NoClassDefFoundError:
>    
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler]; nested:
>    
NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];
>     nested:
>    
ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];
>     ]]
>
>     ...Ken
>
>

(Kenneth Loafman) #6

Never mind... My fault...

I updated the plugin cloud-aws.zip and all is now 'green'.

...Thanks,
...Ken

Kenneth Loafman wrote:

For some reason when I do the git clone, I don't end up with a
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
anywhere.

Here's the steps I took to re-clone Elastisearch:
git clone git://github.com/elasticsearch/elasticsearch.git
./gradlew clean release
unzip build/distributions/elasticsearch-0.9.1-SNAPSHOT.zip
cd elasticsearch-0.9.1-SNAPSHOT/
for N in 0 1 2; do
cp -rp bin lib /opt/es-node$N/
done

Then I restarted via:
for N in 0 1 2; do
/opt/es-node$N/bin/service/elasticsearch restart
done

Do you see anything obviously wrong?

...Ken

Shay Banon wrote:

Did a fresh clone of master, ran "gradle release", opened the
distribution, started elasticsearch, and created an index on it. All
seems to work...

-shay.banon

On Sun, Aug 22, 2010 at 6:12 PM, Kenneth Loafman
<kenneth.loafman@gmail.com mailto:kenneth.loafman@gmail.com> wrote:

Same thing, but maybe a bit longer traceback...

> [10:09:38,606][WARN ][indices.cluster          ] [USAgent]
[google][0] failed to create shard
> org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>         at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>         at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>         at
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>         at
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
>         at
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
>         at
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>         at
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>         at
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
>         at
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
>         at
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
>         at
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
>         at
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
>         at
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
>         at
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
>         at
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
>         at
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
>         at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
>         at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
>         at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
>         at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
>         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>         at java.lang.Class.getDeclaredConstructors0(Native Method)
>         at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
>         at java.lang.Class.getDeclaredConstructors(Class.java:1853)
>         at
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
>         at
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
>         at
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
>         at
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
>         ... 26 more
> Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>         at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>         ... 36 more

...Ken

Shay Banon wrote:
> Can you run a "gradlew clean release" (the clean is important)?
>
> -shay.banon
>
> On Sun, Aug 22, 2010 at 3:20 PM, Kenneth Loafman
> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
<mailto:kenneth.loafman@gmail.com
<mailto:kenneth.loafman@gmail.com>>> wrote:
>
>     I upgraded this morning to the new snapshot and it's continuously
>     dumping while restoring:
>
>     [06:52:04,019][WARN ][indices.cluster          ] [Brown, Abe]
>     [test-index][1] failed to create shard
>     org.elasticsearch.common.inject.internal.ComputationException:
>     java.lang.NoClassDefFoundError:
>     org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>        at
>    
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>        at
>    
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>        at
>    
org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>        at
>    
org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
>        at
>    
org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)
>        at
>    
org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)
>        at
>    
org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)
>        at
>    
org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)
>        at
>    
org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
>        at
>    
org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)
>        at
>    
org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)
>        at
>    
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)
>        at
>    
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)
>        at
>    
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)
>        at
>    
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)
>        at
>    
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
>    
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
>     Caused by: java.lang.NoClassDefFoundError:
>     org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>        at java.lang.Class.getDeclaredConstructors0(Native Method)
>        at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
>        at java.lang.Class.getDeclaredConstructors(Class.java:1853)
>        at
>    
org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)
>        at
>    
org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)
>        at
>    
org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)
>        at
>    
org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)
>        ... 26 more
>     Caused by: java.lang.ClassNotFoundException:
>     org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>        at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>        ... 36 more
>     [06:52:04,019][WARN ][cluster.action.shard     ] [Brown, Abe]
sending
>     failed shard for [test-index][1],
>     node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P],
s[INITIALIZING], reason
>     [Failed to create shard, message
>     [ComputationException[java.lang.NoClassDefFoundError:
>    
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler]; nested:
>    
NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];
>     nested:
>    
ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];
>     ]]
>
>     ...Ken
>
>

(Shay Banon) #7

great. Just a note, you don't have to have several installation of
elasticsearch in order to start several instances, just fire up several
instances from the same installation.

-shay.banon

On Sun, Aug 22, 2010 at 11:02 PM, Kenneth Loafman <kenneth.loafman@gmail.com

wrote:

Never mind... My fault...

I updated the plugin cloud-aws.zip and all is now 'green'.

...Thanks,
...Ken

Kenneth Loafman wrote:

For some reason when I do the git clone, I don't end up with a
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
anywhere.

Here's the steps I took to re-clone Elastisearch:
git clone git://github.com/elasticsearch/elasticsearch.git
./gradlew clean release
unzip build/distributions/elasticsearch-0.9.1-SNAPSHOT.zip
cd elasticsearch-0.9.1-SNAPSHOT/
for N in 0 1 2; do
cp -rp bin lib /opt/es-node$N/
done

Then I restarted via:
for N in 0 1 2; do
/opt/es-node$N/bin/service/elasticsearch restart
done

Do you see anything obviously wrong?

...Ken

Shay Banon wrote:

Did a fresh clone of master, ran "gradle release", opened the
distribution, started elasticsearch, and created an index on it. All
seems to work...

-shay.banon

On Sun, Aug 22, 2010 at 6:12 PM, Kenneth Loafman
<kenneth.loafman@gmail.com mailto:kenneth.loafman@gmail.com> wrote:

Same thing, but maybe a bit longer traceback...

> [10:09:38,606][WARN ][indices.cluster          ] [USAgent]
[google][0] failed to create shard
> org.elasticsearch.common.inject.internal.ComputationException:
java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>         at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)

>         at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)

>         at

org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)

>         at

org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)

>         at

org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)

>         at

org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)

>         at

org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)

>         at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)

>         at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)

>         at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)

>         at

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)

>         at

org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)

>         at

org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)

>         at

org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)

>         at

org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)

>         at

org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)

>         at

org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)

>         at

org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)

>         at

org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)

>         at

org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)

>         at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)

>         at

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)

>         at

org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)

>         at

org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)

>         at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

>         at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.NoClassDefFoundError:
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>         at java.lang.Class.getDeclaredConstructors0(Native Method)
>         at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
>         at

java.lang.Class.getDeclaredConstructors(Class.java:1853)

>         at

org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)

>         at

org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)

>         at

org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)

>         at

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)

>         at

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)

>         at

org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)

>         at

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)

>         ... 26 more
> Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>         at java.security.AccessController.doPrivileged(Native

Method)

>         at

java.net.URLClassLoader.findClass(URLClassLoader.java:205)

>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>         at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>         ... 36 more

...Ken

Shay Banon wrote:
> Can you run a "gradlew clean release" (the clean is important)?
>
> -shay.banon
>
> On Sun, Aug 22, 2010 at 3:20 PM, Kenneth Loafman
> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
<mailto:kenneth.loafman@gmail.com
<mailto:kenneth.loafman@gmail.com>>> wrote:
>
>     I upgraded this morning to the new snapshot and it's

continuously

>     dumping while restoring:
>
>     [06:52:04,019][WARN ][indices.cluster          ] [Brown, Abe]
>     [test-index][1] failed to create shard
>     org.elasticsearch.common.inject.internal.ComputationException:
>     java.lang.NoClassDefFoundError:
>     org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>        at
>

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)

>        at
>

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)

>        at
>

org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)

>        at
>

org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)

>        at
>

org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)

>        at
>

org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:645)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:581)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:572)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:172)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.getBindingOrThrow(InjectorImpl.java:132)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.getInternalFactory(InjectorImpl.java:651)

>        at
>

org.elasticsearch.common.inject.FactoryProxy.notify(FactoryProxy.java:48)

>        at
>

org.elasticsearch.common.inject.BindingProcessor.runCreationListeners(BindingProcessor.java:230)

>        at
>

org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:131)

>        at
>

org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)

>        at
>

org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:144)

>        at
>

org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:265)

>        at
>

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:292)

>        at
>

org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:261)

>        at
>

org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:118)

>        at
>

org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:202)

>        at
>

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

>        at
>

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

>        at java.lang.Thread.run(Thread.java:636)
>     Caused by: java.lang.NoClassDefFoundError:
>     org/elasticsearch/indices/recovery/throttler/RecoveryThrottler
>        at java.lang.Class.getDeclaredConstructors0(Native Method)
>        at
java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
>        at java.lang.Class.getDeclaredConstructors(Class.java:1853)
>        at
>

org.elasticsearch.common.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:185)

>        at
>

org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:61)

>        at
>

org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:31)

>        at
>

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:39)

>        at
>

org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:35)

>        at
>

org.elasticsearch.common.inject.internal.FailableCache$1.apply(FailableCache.java:35)

>        at
>

org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:549)

>        ... 26 more
>     Caused by: java.lang.ClassNotFoundException:
>     org.elasticsearch.indices.recovery.throttler.RecoveryThrottler
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>        at java.security.AccessController.doPrivileged(Native

Method)

>        at

java.net.URLClassLoader.findClass(URLClassLoader.java:205)

>        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>        at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>        ... 36 more
>     [06:52:04,019][WARN ][cluster.action.shard     ] [Brown, Abe]
sending
>     failed shard for [test-index][1],
>     node[df40085c-0242-4fd0-b0a2-e202f3ed4696], [P],
s[INITIALIZING], reason
>     [Failed to create shard, message
>     [ComputationException[java.lang.NoClassDefFoundError:
>
org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];

nested:

>

NoClassDefFoundError[org/elasticsearch/indices/recovery/throttler/RecoveryThrottler];

>     nested:
>

ClassNotFoundException[org.elasticsearch.indices.recovery.throttler.RecoveryThrottler];

>     ]]
>
>     ...Ken
>
>

(system) #8