IndexShardGatewaySnapshotFailedException caused by StringIndexOutOfBoundsException during indexing


(Franz Allan Valencia See) #1

While indexing, I sometimes get this, and once this happens, my log will be
full of this, and my indexing will slow down significantly (using
elasticsearch 0.9.0).

What does this mean and how can I fix it?

[14:49:43,014][WARN ][index.gateway ] [Slade, Frederick]
[application][1] failed to snapshot (scheduled)
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[application][1] Failed to snapshot
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:243)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$SnapshotRunnable.run(IndexShardGatewayService.java:295)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
range: 32
at java.lang.String.checkBounds(String.java:401)
at java.lang.String.(String.java:482)
at org.elasticsearch.common.Digest.md5HexFromByteArray(Digest.java:210)
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.buildVirtualBlobs(BlobStoreIndexShardGateway.java:760)
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:167)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:211)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:206)
at
org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:378)
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:206)

Thanks,

Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see


(Shay Banon) #2

Hi,

Seems like the md5 was not written properly beforehand, and the gateway
should simple ignore it, but it doesn't (it ignores io based exceptions, not
cases when the size read is not enough). Pushed a fix for it to master.
Which gateway are you using?

-shay.banon

On Thu, Jul 29, 2010 at 8:52 AM, Franz Allan Valencia See <
franz.see@gmail.com> wrote:

While indexing, I sometimes get this, and once this happens, my log will be
full of this, and my indexing will slow down significantly (using
elasticsearch 0.9.0).

What does this mean and how can I fix it?

[14:49:43,014][WARN ][index.gateway ] [Slade, Frederick]
[application][1] failed to snapshot (scheduled)
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[application][1] Failed to snapshot
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:243)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$SnapshotRunnable.run(IndexShardGatewayService.java:295)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
range: 32
at java.lang.String.checkBounds(String.java:401)
at java.lang.String.(String.java:482)
at org.elasticsearch.common.Digest.md5HexFromByteArray(Digest.java:210)
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.buildVirtualBlobs(BlobStoreIndexShardGateway.java:760)
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:167)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:211)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:206)
at
org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:378)
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:206)

Thanks,

Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see


(Franz Allan Valencia See) #3

hdfs gateway

On Thu, Jul 29, 2010 at 8:11 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

Seems like the md5 was not written properly beforehand, and the gateway
should simple ignore it, but it doesn't (it ignores io based exceptions, not
cases when the size read is not enough). Pushed a fix for it to master.
Which gateway are you using?

-shay.banon

On Thu, Jul 29, 2010 at 8:52 AM, Franz Allan Valencia See <
franz.see@gmail.com> wrote:

While indexing, I sometimes get this, and once this happens, my log will
be full of this, and my indexing will slow down significantly (using
elasticsearch 0.9.0).

What does this mean and how can I fix it?

[14:49:43,014][WARN ][index.gateway ] [Slade, Frederick]
[application][1] failed to snapshot (scheduled)
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[application][1] Failed to snapshot
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:243)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$SnapshotRunnable.run(IndexShardGatewayService.java:295)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
range: 32
at java.lang.String.checkBounds(String.java:401)
at java.lang.String.(String.java:482)
at
org.elasticsearch.common.Digest.md5HexFromByteArray(Digest.java:210)
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.buildVirtualBlobs(BlobStoreIndexShardGateway.java:760)
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:167)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:211)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:206)
at
org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:412)
at
org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:378)
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:206)

Thanks,

Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see


(system) #4