# Long GC Pauses

**URL:** <https://discuss.elastic.co/t/long-gc-pauses/14682>\
**Category:** Elasticsearch\
**Created:** [December 3, 2013, 3:52pm UTC](https://discuss.elastic.co/t/long-gc-pauses/14682 "2013-12-03T15:52:13Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\
**Post date:** [December 3, 2013, 3:52pm UTC](https://discuss.elastic.co/t/long-gc-pauses/14682/1 "2013-12-03T15:52:13Z")

</div>

Hi all,

There are 50 indexes, each contains 3 primary shards and 1 replica. Some  
threads running every 15 minutes to search and index a few documents(each  
thread process at max 10 docs).

After some days, ES get into long GC pauses and at the end split brain  
problem.

From the bigdesk we could see that more than 60% of heap is not used.

The only thing which we think could be a problem is many manually refresh  
calls, but I'm not sure.

4 GB assigned to ES process, xms and xmx are set equally.

Total system memory is 12GB

3 other java processes using almost 4 GB.

ES version: 90.3

Java version: 1.7.0\_25

Es vm configuration:

JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"

JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"

JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"

JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"

JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCondCardMark"

JAVA\_OPTS="$JAVA\_OPTS -XX:+UseTLAB"

JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSClassUnloadingEnabled"

JAVA\_OPTS="$JAVA\_OPTS -XX:MaxGCPauseMillis=10000"

ES configuration:

ndex.cache.filter.max\_size: 10

index.store.throttle.type: merge

index.compound\_format: false

index.cache.field.expire: 1m

index.merge.policy.merge\_factor: 30

index.cache.filter.expire: 1m

index.refresh\_interval: -1

index.number\_of\_replicas: 1

index.version.created: 200599

index.store.throttle.max\_bytes\_per\_sec: 5mb

index.number\_of\_shards: 3

index.translog.flush\_threshold\_period: 60s

index.merge.policy.use\_compound\_file: false

index.store.compress.stored: true

index.cache.field.type: resident

index.indices.memory.index\_buffer\_size: 20%

bootstrap.mlockall is not configured yet, but I think there is no problem  
with memory swapping atm.

Can someone help?

Thanks in advance,

Vahid

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/51a19985-4f36-44af-b4dd-b2fb27556717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51a19985-4f36-44af-b4dd-b2fb27556717%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\
**Post date:** [December 3, 2013, 4:13pm UTC](https://discuss.elastic.co/t/long-gc-pauses/14682/2 "2013-12-03T16:13:55Z")

</div>

Hi Vahid,

I can't say what your problem is (maybe someone else has an insight - all  
your settings look fine to me), but here are some "leads":

- it would be interesting to know if switching to the G1 garbage collector  
would help
- maybe upgrading your JVM would help, even though yours is pretty fresh
- it would be interesting to see how your memory pool and garbage  
collection is doing over time. SPM for  
Elasticsearch[http://sematext.com/spm/elasticsearch-performance-monitoring/](http://sematext.com/spm/elasticsearch-performance-monitoring/)can  
help you with that, and there's a free plan that should be good enough  
for diagnostics. With this information, you'll probably be able to tune  
your GC settings for shorter pauses (maybe share some graphs here and I'm  
sure someone will give you useful hints)

Best regards,  
Radu

On Tue, Dec 3, 2013 at 5:52 PM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote:

> Hi all,
> 
> There are 50 indexes, each contains 3 primary shards and 1 replica. Some  
> threads running every 15 minutes to search and index a few documents(each  
> thread process at max 10 docs).
> 
> After some days, ES get into long GC pauses and at the end split brain  
> problem.
> 
> From the bigdesk we could see that more than 60% of heap is not used.
> 
> The only thing which we think could be a problem is many manually refresh  
> calls, but I'm not sure.
> 
> 4 GB assigned to ES process, xms and xmx are set equally.
> 
> Total system memory is 12GB
> 
> 3 other java processes using almost 4 GB.
> 
> ES version: 90.3
> 
> Java version: 1.7.0\_25
> 
> Es vm configuration:
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCondCardMark"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+UseTLAB"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSClassUnloadingEnabled"
> 
> JAVA\_OPTS="$JAVA\_OPTS -XX:MaxGCPauseMillis=10000"
> 
> ES configuration:
> 
> ndex.cache.filter.max\_size: 10
> 
> index.store.throttle.type: merge
> 
> index.compound\_format: false
> 
> index.cache.field.expire: 1m
> 
> index.merge.policy.merge\_factor: 30
> 
> index.cache.filter.expire: 1m
> 
> index.refresh\_interval: -1
> 
> index.number\_of\_replicas: 1
> 
> index.version.created: 200599
> 
> index.store.throttle.max\_bytes\_per\_sec: 5mb
> 
> index.number\_of\_shards: 3
> 
> index.translog.flush\_threshold\_period: 60s
> 
> index.merge.policy.use\_compound\_file: false
> 
> index.store.compress.stored: true
> 
> index.cache.field.type: resident
> 
> index.indices.memory.index\_buffer\_size: 20%
> 
> bootstrap.mlockall is not configured yet, but I think there is no problem  
> with memory swapping atm.
> 
> Can someone help?
> 
> Thanks in advance,
> 
> Vahid
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/51a19985-4f36-44af-b4dd-b2fb27556717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51a19985-4f36-44af-b4dd-b2fb27556717%40googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
Performance Monitoring \* Log Analytics \* Search Analytics  
Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_2X6VkU2wE3SoLzy6RzOrLfynXFKyVKdHAQTPKvjY85yw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_2X6VkU2wE3SoLzy6RzOrLfynXFKyVKdHAQTPKvjY85yw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 4, 2013, 7:56am UTC](https://discuss.elastic.co/t/long-gc-pauses/14682/3 "2013-12-04T07:56:25Z")

</div>

Hey,

If you total system memory is 12GB, you have 4GB of heap, 4GB of other java  
process, there are only 4GB left for the file system cache. This is pretty  
easy to fill, if you are doing quite some searches on that machine. So this  
makes setting bootstrap.mlockall crucial. If a garbage collection has to go  
to disk/swap in order to collect garbage, I am not surprised it is very  
slow.

Are there any specific reasons you set all this additional JVM setups?  
There is no dynamic JVM language involved so why call with  
CMSClassUnloadingEnabled? Can you try with the defaults first, before  
tuning in order to eliminate those as a source of these problems? Same goes  
for pause times, and tlab thread allocation. I dont see a special setup  
here, why the standard settings should be a bad choice.

Also, upgrading your JVM should be postponed, as newer versions have  
problems with Lucene which are not fixed yet, I would stay with the  
current. I would not recommend using G1, but you are free to try - there  
are people telling about speedups but there are at least as many people  
telling about JVM crashes 🙂

Last, using nodes stats and graphing the output might make sense here, take  
a special view at fielddata or maybe there is another part of the heap  
space under pressure than the oldgen pool. See

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

--Alex

On Tue, Dec 3, 2013 at 5:13 PM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)wrote:

> Hi Vahid,
> 
> I can't say what your problem is (maybe someone else has an insight - all  
> your settings look fine to me), but here are some "leads":
> 
> - it would be interesting to know if switching to the G1 garbage collector  
> would help
> - maybe upgrading your JVM would help, even though yours is pretty fresh
> - it would be interesting to see how your memory pool and garbage  
> collection is doing over time. SPM for Elasticsearch[http://sematext.com/spm/elasticsearch-performance-monitoring/](http://sematext.com/spm/elasticsearch-performance-monitoring/)can help you with that, and there's a free plan that should be good enough  
> for diagnostics. With this information, you'll probably be able to tune  
> your GC settings for shorter pauses (maybe share some graphs here and I'm  
> sure someone will give you useful hints)
> 
> Best regards,  
> Radu
> 
> On Tue, Dec 3, 2013 at 5:52 PM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote:
> 
> > Hi all,
> > 
> > There are 50 indexes, each contains 3 primary shards and 1 replica. Some  
> > threads running every 15 minutes to search and index a few documents(each  
> > thread process at max 10 docs).
> > 
> > After some days, ES get into long GC pauses and at the end split brain  
> > problem.
> > 
> > From the bigdesk we could see that more than 60% of heap is not used.
> > 
> > The only thing which we think could be a problem is many manually refresh  
> > calls, but I'm not sure.
> > 
> > 4 GB assigned to ES process, xms and xmx are set equally.
> > 
> > Total system memory is 12GB
> > 
> > 3 other java processes using almost 4 GB.
> > 
> > ES version: 90.3
> > 
> > Java version: 1.7.0\_25
> > 
> > Es vm configuration:
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseParNewGC"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseConcMarkSweepGC"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:CMSInitiatingOccupancyFraction=75"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseCondCardMark"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+UseTLAB"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:+CMSClassUnloadingEnabled"
> > 
> > JAVA\_OPTS="$JAVA\_OPTS -XX:MaxGCPauseMillis=10000"
> > 
> > ES configuration:
> > 
> > ndex.cache.filter.max\_size: 10
> > 
> > index.store.throttle.type: merge
> > 
> > index.compound\_format: false
> > 
> > index.cache.field.expire: 1m
> > 
> > index.merge.policy.merge\_factor: 30
> > 
> > index.cache.filter.expire: 1m
> > 
> > index.refresh\_interval: -1
> > 
> > index.number\_of\_replicas: 1
> > 
> > index.version.created: 200599
> > 
> > index.store.throttle.max\_bytes\_per\_sec: 5mb
> > 
> > index.number\_of\_shards: 3
> > 
> > index.translog.flush\_threshold\_period: 60s
> > 
> > index.merge.policy.use\_compound\_file: false
> > 
> > index.store.compress.stored: true
> > 
> > index.cache.field.type: resident
> > 
> > index.indices.memory.index\_buffer\_size: 20%
> > 
> > bootstrap.mlockall is not configured yet, but I think there is no problem  
> > with memory swapping atm.
> > 
> > Can someone help?
> > 
> > Thanks in advance,
> > 
> > Vahid
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/51a19985-4f36-44af-b4dd-b2fb27556717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/51a19985-4f36-44af-b4dd-b2fb27556717%40googlegroups.com)  
> > .  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> Performance Monitoring \* Log Analytics \* Search Analytics  
> Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/)
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_2X6VkU2wE3SoLzy6RzOrLfynXFKyVKdHAQTPKvjY85yw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_2X6VkU2wE3SoLzy6RzOrLfynXFKyVKdHAQTPKvjY85yw%40mail.gmail.com)  
> .
> 
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-aPr2%2B8Qu%2BtSbOLR4Yfbg-cBdncrL%2BQsbWXTq7zodzBQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-aPr2%2B8Qu%2BtSbOLR4Yfbg-cBdncrL%2BQsbWXTq7zodzBQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\
**Post date:** [December 4, 2013, 10:44am UTC](https://discuss.elastic.co/t/long-gc-pauses/14682/4 "2013-12-04T10:44:11Z")

</div>

Hi,

> Many thanks Radu and Alex for your replies,

Atm I'm not granted to install any application on customer system, so using  
SPM for me is not an option.  
I've created a screenshot of one of the nodes, maybe it give you more info.

Best regards,  
Vahid

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/32c8f2ef-57d2-4461-a977-0d90d2ed19e8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/32c8f2ef-57d2-4461-a977-0d90d2ed19e8%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\
**Post date:** [December 4, 2013, 10:51am UTC](https://discuss.elastic.co/t/long-gc-pauses/14682/5 "2013-12-04T10:51:21Z")

</div>

On this cluster(which graphs provided) bootstrap.mlockall=true is  
configured and from the top command I see swap memory used is 0.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/81aecc2d-78cc-45a7-b18c-6bd08ceadeb9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/81aecc2d-78cc-45a7-b18c-6bd08ceadeb9%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:03am UTC](https://discuss.elastic.co/t/long-gc-pauses/14682/6 "2017-07-06T02:03:25Z")

</div>