ES starting process and using huge HEAP memory problem


(prometheus) #1

Last couple of days I am looking the forums and the blogs regarding to find
a help or clue about the Large indexes and the Heap usage similar to the
our use case.

Unfortunatelly I didn’t find a solution that helps my case. Here the
setup:

We have two test server, each :

128 gig Ram
24 core xeon cpu
3TB disk

We have 10 indexes and 5 shards 0 replica . Each index has around 120 gig
size. Two of them 190 Gig size. Index mapping has a parent child
relation.

ES heap is 80 gig

Our main problem is the opening ES server. When ES start to open indexes
it always require recovering process even we decently close the ES server
before.

When ES recovering the index shard Heap always going up. Specially when ES
start to recover/initialize 190 gig size index Heap almost full and its
going to endless GC process .

Why ES using so much heap to open / recover / initialize shards ?

After successfully opened shards ES not releasing heap that used ?

What is the mechanism behind the initializing indexes process ?

Why ES recovering indexes every time ?

What could be your suggestion ?

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c1a453c-04d5-4f99-ab0a-d9bd743e3b34%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(prometheus) #2

I forgot to mention ES version is 0.90.8

On Wednesday, December 25, 2013 3:13:12 PM UTC+2, Prometheus WillSurvive
wrote:

Last couple of days I am looking the forums and the blogs regarding to
find a help or clue about the Large indexes and the Heap usage similar to
the our use case.

Unfortunatelly I didn’t find a solution that helps my case. Here the
setup:

We have two test server, each :

128 gig Ram
24 core xeon cpu
3TB disk

We have 10 indexes and 5 shards 0 replica . Each index has around 120 gig
size. Two of them 190 Gig size. Index mapping has a parent child
relation.

ES heap is 80 gig

Our main problem is the opening ES server. When ES start to open indexes
it always require recovering process even we decently close the ES server
before.

When ES recovering the index shard Heap always going up. Specially when ES
start to recover/initialize 190 gig size index Heap almost full and its
going to endless GC process .

Why ES using so much heap to open / recover / initialize shards ?

After successfully opened shards ES not releasing heap that used ?

What is the mechanism behind the initializing indexes process ?

Why ES recovering indexes every time ?

What could be your suggestion ?

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/15b79f2e-3c55-4e61-a07a-2f4dfc77bf50%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #3

Running > 32GB for heap size brings a lot of inefficiencies into play as
java pointers will not be compressed and your GC will increase (as you can
see).

When you shut a node down the indexes it has allocated to it go into an
unallocated state and the cluster will try to reallocate all of these
shards. But if you (re)join a node to the cluster it will still initialise
and reallocate them even if it only ends up putting them onto the same
node. This is to ensure the cluster state is maintained and your shards are
evenly distributed.

If you are just restarting a node for whatever reason, you can
set cluster.routing.allocation.disable_allocation to true, then restart the
node, and when it comes back it will simply reinitialise and reopen the
index shards on the local machine, which is a lot faster recovery. Make
sure you set that back to false when things are green again.

Ideally you need to add more nodes to your cluster, you could split those
two and make 4 VMs/containers, or just add more physical nodes. Just
remember my first comment though.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 December 2013 00:13, Prometheus WillSurvive <
prometheus.willsurvive@gmail.com> wrote:

Last couple of days I am looking the forums and the blogs regarding to
find a help or clue about the Large indexes and the Heap usage similar to
the our use case.

Unfortunatelly I didn’t find a solution that helps my case. Here the
setup:

We have two test server, each :

128 gig Ram
24 core xeon cpu
3TB disk

We have 10 indexes and 5 shards 0 replica . Each index has around 120 gig
size. Two of them 190 Gig size. Index mapping has a parent child
relation.

ES heap is 80 gig

Our main problem is the opening ES server. When ES start to open indexes
it always require recovering process even we decently close the ES server
before.

When ES recovering the index shard Heap always going up. Specially when ES
start to recover/initialize 190 gig size index Heap almost full and its
going to endless GC process .

Why ES using so much heap to open / recover / initialize shards ?

After successfully opened shards ES not releasing heap that used ?

What is the mechanism behind the initializing indexes process ?

Why ES recovering indexes every time ?

What could be your suggestion ?

thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5c1a453c-04d5-4f99-ab0a-d9bd743e3b34%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aSGCdkEm3%3D-xcA5sKFWkHnsd5ZdreK7VVoOgF99btW8A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(prometheus) #4

Hi Mr. Walkom,

thank you for your kind response.

Actually we already set cluster.routing.allocation.disable_allocation to
true and there is no shard routing between nodes to avoid this issue.
When indexes start to open its using gigs of memory . If I set the Heap
below 32 gig its not going to open and keeping GC running and deadlock. I
suspected that ES loading something to the cache when its start opening.
Below a capture from when the index opening threads:


::: [SE03indicto][57GE4QrOQV-w-5OzVcd6mQ][inet[/10.0.0.63:9300]]

100.5% (502.7ms out of 500ms) cpu usage by thread
'elasticsearch[SE03indicto][warmer][T#6]'
3/10 snapshots sharing following 18 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:169)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:271)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:137)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:113)
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.readTermsBlock(Lucene41PostingsReader.java:215)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2410)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadNextFloorBlock(BlockTreeTermsReader.java:2340)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.next(BlockTreeTermsReader.java:2115)
org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredTermsEnum.next(BloomFilterPostingsFormat.java:257)
org.elasticsearch.index.cache.id.simple.SimpleIdCache.refresh(SimpleIdCache.java:151)
org.elasticsearch.search.SearchService$FieldDataWarmer$2.run(SearchService.java:689)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
4/10 snapshots sharing following 8 elements
org.elasticsearch.common.hppc.ObjectIntOpenHashMap.containsKey(ObjectIntOpenHashMap.java:702)
org.elasticsearch.index.cache.id.simple.SimpleIdCache$TypeBuilder.canReuse(SimpleIdCache.java:323)
org.elasticsearch.index.cache.id.simple.SimpleIdCache.checkIfCanReuse(SimpleIdCache.java:287)
org.elasticsearch.index.cache.id.simple.SimpleIdCache.refresh(SimpleIdCache.java:181)
org.elasticsearch.search.SearchService$FieldDataWarmer$2.run(SearchService.java:689)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
2/10 snapshots sharing following 17 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:169)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:271)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:137)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:113)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2384)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadNextFloorBlock(BlockTreeTermsReader.java:2340)
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.next(BlockTreeTermsReader.java:2115)
org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredTermsEnum.next(BloomFilterPostingsFormat.java:257)
org.elasticsearch.index.cache.id.simple.SimpleIdCache.refresh(SimpleIdCache.java:151)
org.elasticsearch.search.SearchService$FieldDataWarmer$2.run(SearchService.java:689)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.Thread


What I found a manual solution is clearing caches while indexes opening so
heap not going crazy. I also tried the 0.90.9 that a new feature :
index.codec.bloom.load that can control if the boom filters will be loaded
or not. But 0.90.9 brought us to cluster inconsistency and can not open
indexes.

I would be very happy to use HEAP under 32 gig. How can I say the ES just
initialize the index and not load any cache assuming this is the problem.

We also used solr for a couple of years and very big index sizes. It
never took such a long time and huge memory usage. Keep digging the issue
and keep posted.

Thanks

On Wednesday, December 25, 2013 11:25:09 PM UTC+2, Mark Walkom wrote:

Running > 32GB for heap size brings a lot of inefficiencies into play as
java pointers will not be compressed and your GC will increase (as you can
see).

When you shut a node down the indexes it has allocated to it go into an
unallocated state and the cluster will try to reallocate all of these
shards. But if you (re)join a node to the cluster it will still initialise
and reallocate them even if it only ends up putting them onto the same
node. This is to ensure the cluster state is maintained and your shards are
evenly distributed.

If you are just restarting a node for whatever reason, you can
set cluster.routing.allocation.disable_allocation to true, then restart the
node, and when it comes back it will simply reinitialise and reopen the
index shards on the local machine, which is a lot faster recovery. Make
sure you set that back to false when things are green again.

Ideally you need to add more nodes to your cluster, you could split those
two and make 4 VMs/containers, or just add more physical nodes. Just
remember my first comment though.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 26 December 2013 00:13, Prometheus WillSurvive <
prometheus....@gmail.com <javascript:>> wrote:

Last couple of days I am looking the forums and the blogs regarding to
find a help or clue about the Large indexes and the Heap usage similar to
the our use case.

Unfortunatelly I didn’t find a solution that helps my case. Here the
setup:

We have two test server, each :

128 gig Ram
24 core xeon cpu
3TB disk

We have 10 indexes and 5 shards 0 replica . Each index has around 120
gig size. Two of them 190 Gig size. Index mapping has a parent child
relation.

ES heap is 80 gig

Our main problem is the opening ES server. When ES start to open indexes
it always require recovering process even we decently close the ES server
before.

When ES recovering the index shard Heap always going up. Specially when
ES start to recover/initialize 190 gig size index Heap almost full and its
going to endless GC process .

Why ES using so much heap to open / recover / initialize shards ?

After successfully opened shards ES not releasing heap that used ?

What is the mechanism behind the initializing indexes process ?

Why ES recovering indexes every time ?

What could be your suggestion ?

thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5c1a453c-04d5-4f99-ab0a-d9bd743e3b34%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f0f6996-f5f0-4ec1-b9ae-4831c5e67959%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #5

Do you use parent/child feature?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF_HQ0iXRsuaKyHJsSp-8ZkFaDe53rmxYOTcV6X_aLtnw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(prometheus) #6

Yes we are using parent child relation in our indexes..

What I discovered tonight that *ID Cache Size: growing Gigabytes
(generally %10 percent of the shard size). I am clearing the cache while
opening the indexes repeatedly so I can manage the open all indexes with
in 30 Gb HEAP size. *

This is the latest situation.

Thnx

On Thursday, December 26, 2013 9:39:33 PM UTC+2, Jörg Prante wrote:

Do you use parent/child feature?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ff94d64d-c113-44f2-8ffb-df6fb7efb0d2%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #7

All parent IDs are loaded into cache by default. Try the setting

index.cache.id.reuse: true

It should save some memory.

Another method to save cache resources is using an optimized index, but
this hint is not useful when you can't start ES.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_NC3h-c2_rASD5eBYUmSD71F_Jv90m_EF%3DecjoeCUtA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Karol Gwaj) #8

actually the setting is: index.cache.id.simple.reuse: true
(just in case if you still having problem with heap memory)

On Friday, December 27, 2013 2:01:57 PM UTC, Jörg Prante wrote:

All parent IDs are loaded into cache by default. Try the setting

index.cache.id.reuse: true

It should save some memory.

Another method to save cache resources is using an optimized index, but
this hint is not useful when you can't start ES.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a0ef280b-019e-43c9-b91d-82448e0ecd2b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #9