Hey, Igor, thanks for answering! and sorry for the delay. Didn't catch the
update.
I explain:
- we have one cluster of one machine which is only meant for serving
search requests. the goal is not to index anything to it. It contains 1.7k
indices, give it or take it.
- every day, those 1.7k indices are reindexed, and snapshoted in pairs
to a S3 repository (producint 850 snapshots)repository.
- every day, the one "reading only" cluster of the first point restores
those 850 snapshots to "update" its 1.7k indices from that same S3
repository.
It works like a real charm. Load has dropped dramatically, and we can set a
"farm" of temporary machines to do the indexing duties.
But memory consumption never stops growing.
we don't get any "out of memory" error or anything. In fact, there is
nothing in the logs that shows any error, but after a week or a few days,
the host has its memory almost exhausted and elasticsearch is not
responding. The memory consumption is of course way ahead of the HEAP_SIZE
We have to restart it and, when we do it we get the following error:
java.util.concurrent.RejectedExecutionException: Worker has already been
shutdown
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.
executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.
executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.
fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.
exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(
DefaultChannelPipeline.java:781)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.
java:725)
at org.elasticsearch.common.netty.handler.codec.oneone.
OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at org.elasticsearch.common.netty.handler.codec.oneone.
OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:591)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.
java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.
java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(
AbstractChannel.java:248)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(
NettyHttpChannel.java:158)
at org.elasticsearch.rest.action.search.RestSearchAction$1.
onResponse(RestSearchAction.java:106)
at org.elasticsearch.rest.action.search.RestSearchAction$1.
onResponse(RestSearchAction.java:98)
at org.elasticsearch.action.search.type.
TransportSearchQueryAndFetchAction$AsyncAction.innerFinishHim(
TransportSearchQueryAndFetchAction.java:94)
at org.elasticsearch.action.search.type.
TransportSearchQueryAndFetchAction$AsyncAction.moveToSecondPhase(
TransportSearchQueryAndFetchAction.java:77)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase(
TransportSearchTypeAction.java:425)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(
TransportSearchTypeAction.java:243)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$3.onResult(
TransportSearchTypeAction.java:219)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$3.onResult(
TransportSearchTypeAction.java:216)
at org.elasticsearch.search.action.SearchServiceTransportAction.
sendExecuteFetch(SearchServiceTransportAction.java:305)
at org.elasticsearch.action.search.type.
TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(
TransportSearchQueryAndFetchAction.java:71)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.java:216)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(
TransportSearchTypeAction.java:203)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.
java:186)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
It looks like it shutdowns itself down for some reason...
The hosts in which elasticsearch is living has nothing else installed
besides a standard ubuntu distribution. It's completely devoted to
elasticsearch.
the memory consumption grows a 10% in 36h
On Monday, June 30, 2014 10:45:18 PM UTC-4, Igor Motov wrote:
Just to make sure I got it right, you really meant 700 restores (not just
700 snapshots), correct? What type of repository are you using? Could you
add a bit more details about your use case?
On Monday, June 30, 2014 8:53:10 AM UTC-4, JoeZ99 wrote:
We have one one-machine cluster with about 1k indices. It used to work
flawlessly , (being with a high load, of course)
but since we started to use heavily the snapshot-restore feature, it's
getting its memory exhausted within 7 days of use. The cluster make about
700 restore proceedings during the day. Maybe there are some memory
considerations when using the restore feature???
--
uh, oh http://www.youtube.com/watch?v=GMD_T7ICL0o.
http://www.defectivebydesign.org/no-drm-in-html5
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3efafd17-5a2b-4dc5-b935-4a0c5c1a8bab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.