OutOfMemoryError when shutting down a gateway


(Cédric) #1

Hi again,

I have a doubt about how things work in ElasticSearch : I've shut down
a node for which a gateway configuration was set. However, when
shutting down the gateway, I saw the following messages in the log.
For me "warn" means that it is not important, although "outofmemory"
generally means "failure" for me. Is it that ElasticSearch will try
another method (no memory mapped files) for storing the index in that
case, or will I loose part of my index ?

[12:10:29,221][WARN ][index.gateway ] [Tana Nile][news4][3]
Failed to snapshot (scheduled)
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[news4][3] Failed to perform snapshot (index files)
at
org.elasticsearch.index.gateway.fs.FsIndexShardGateway.snapshot(FsIndexShardGateway.java:
185)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.snapshot(IndexShardGatewayService.java:179)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.snapshot(IndexShardGatewayService.java:175)
at
org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:
364)
at
org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:
377)
at
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:
175)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$SnapshotRunnable.run(IndexShardGatewayService.java:257)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
441)
at java.util.concurrent.FutureTask
$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by:
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[news4][3] Failed to copy to [/home/cedric/DEV/elasticsearch-0.8.0/
work/elasticsearch/gateway/indices/news4/3/index/_1l.cfs], from dir
[org.elasticsearch.index.store.fs.NioFsStore$CustomNioFSDirectory@/
home/cedric/DEV/elasticsearch-0.8.0/work/elasticsearch/indices/
05229c90-e20a-465e-827d-a5b1f2af05a8/news4/3/index] and file [_1l.cfs]
at org.elasticsearch.index.gateway.fs.FsIndexShardGateway
$1.run(FsIndexShardGateway.java:172)
... 3 more
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
at
sun.nio.ch.FileChannelImpl.transferFromFileChannel(FileChannelImpl.java:
537)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:600)
at
org.elasticsearch.util.io.FileSystemUtils.copyFile(FileSystemUtils.java:
102)
at
org.elasticsearch.util.lucene.Directories.copyFromDirectory(Directories.java:
88)
at org.elasticsearch.index.gateway.fs.FsIndexShardGateway
$1.run(FsIndexShardGateway.java:170)
... 3 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
... 8 more

Thanks,


(Shay Banon) #2

In the upcoming 0.9 I removed completely the native transfer from
elasticsearch as its so buggy in Java. In 0.8 you can disable it by setting
index.gateway.fs.native_copy to false.

-shay.banon

On Thu, Jul 1, 2010 at 1:16 PM, Cédric cedric.champeau@gmail.com wrote:

Hi again,

I have a doubt about how things work in ElasticSearch : I've shut down
a node for which a gateway configuration was set. However, when
shutting down the gateway, I saw the following messages in the log.
For me "warn" means that it is not important, although "outofmemory"
generally means "failure" for me. Is it that ElasticSearch will try
another method (no memory mapped files) for storing the index in that
case, or will I loose part of my index ?

[12:10:29,221][WARN ][index.gateway ] [Tana Nile][news4][3]
Failed to snapshot (scheduled)
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[news4][3] Failed to perform snapshot (index files)
at

org.elasticsearch.index.gateway.fs.FsIndexShardGateway.snapshot(FsIndexShardGateway.java:
185)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.snapshot(IndexShardGatewayService.java:179)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.snapshot(IndexShardGatewayService.java:175)
at
org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:
364)
at

org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:
377)
at

org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:
175)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$SnapshotRunnable.run(IndexShardGatewayService.java:257)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
441)
at java.util.concurrent.FutureTask
$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by:
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:
[news4][3] Failed to copy to [/home/cedric/DEV/elasticsearch-0.8.0/
work/elasticsearch/gateway/indices/news4/3/index/_1l.cfs], from dir
[org.elasticsearch.index.store.fs.NioFsStore$CustomNioFSDirectory@/
home/cedric/DEV/elasticsearch-0.8.0/work/elasticsearch/indices/
05229c90-e20a-465e-827d-a5b1f2af05a8/news4/3/index] and file [_1l.cfs]
at org.elasticsearch.index.gateway.fs.FsIndexShardGateway
$1.run(FsIndexShardGateway.java:172)
... 3 more
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
at
sun.nio.ch.FileChannelImpl.transferFromFileChannel(FileChannelImpl.java:
537)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:600)
at
org.elasticsearch.util.io.FileSystemUtils.copyFile(FileSystemUtils.java:
102)
at

org.elasticsearch.util.lucene.Directories.copyFromDirectory(Directories.java:
88)
at org.elasticsearch.index.gateway.fs.FsIndexShardGateway
$1.run(FsIndexShardGateway.java:170)
... 3 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
... 8 more

Thanks,


(system) #3