# Anyone have G1 GC working? What environment/configs?
**URL:** https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274
**Category:** Elasticsearch
**Created:** [August 20, 2013, 5:02pm UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274 "2013-08-20T17:02:38Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Nathan\_Dial](https://discuss.elastic.co/u/Nathan_Dial)
#### Post date: [August 20, 2013, 5:02pm UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/1 "2013-08-20T17:02:38Z")
I'm trying to tune ES for predictable response times under an update-heavy
load. Our big challenge has been long GC pauses (sometimes stopping the app
for up to 30 seconds).
This post
([http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html))
by Jörg Prante has a lot of knowledgable advice on GC tuning and seems to
endorse the G1 GC for the situation I'm working with. It appears to reduces
long pauses at the expense of higher CPU load, which sounds like just the
trade-off I want to make.
But when I try to run the G1 GC, it causes the JVM to segfault, pretty soon
after starting. I've asked around with a few others in #elasticsearch, and
it seems others are having similar issues with G1--many I've spoken to have
tried it but ditched it when it segfaulted too much.
However, everyone seems to "know somebody" or know of someone who has had
success with G1.
So ... is G1 working for anyone? If you have G1 working and no segfault
issues, can you tell me a little more about your environment? (what OS,
JVM, ES version, JVM flags, app settings etc.) It doesn't make sense to me
that it is segfaulting and dying for me and others, but working okay for
some. There has to be a reason, _if it is_ working for some--I'm growing
skeptical that G1GC is working for anyone until I hear otherwise 🙂
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [August 20, 2013, 9:10pm UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/2 "2013-08-20T21:10:21Z")
I tested the G1GC option a while ago, because I had to reduce the "stop the
world" GC phases on large heaps for my workload. This is a very urgent
issue for me because I want to handle machines with 64g RAM in production.
Back in April, I did not encounter any segfaults, with JDK 1.7.0\_11 or
1.8.0-ea-b82 and ES 0.90.0.RC1. The workload on a single node was ~10k docs
per second bulk indexing, mixed with plain term queries at ~800 qps. No
facet queries or filters.
In the meantime I learned that trove4j crashes randomly on all known JVM
releases 1.7.0\_17+ and 1.8.0+ with G1GC enabled, and I could reproduce it.
I did not try older JVM versions though. Especially the trove4j Maps seem
affected. ES makes use of them heavily in the faceting module so this
explains immediate segfaults when G1GC is enabled for these kind of
workloads. Right now I'm experimenting with HPPC as a trove4j replacement
in ES but I'm not finished yet. And I still have hope that G1GC can somehow
be fixed for the trove4j issue.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Runar\_Myklebust](https://discuss.elastic.co/u/Runar_Myklebust)
#### Post date: [August 21, 2013, 6:45am UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/3 "2013-08-21T06:45:46Z")
We've had no luck with G1 because of the trove4j issue, tried on several
installations on different platforms.
Instead, this seems to be the best solution we have got so far using big
heaps:
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=30
-XX:+UseCMSInitiatingOccupancyOnly
-XX:NewRatio=4
On Tue, Aug 20, 2013 at 7:02 PM, Nathan Dial [nathan.dial@gmail.com](mailto:nathan.dial@gmail.com) wrote:
> I'm trying to tune ES for predictable response times under an update-heavy
> load. Our big challenge has been long GC pauses (sometimes stopping the app
> for up to 30 seconds).
>
> This post (
> [http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html))
> by Jörg Prante has a lot of knowledgable advice on GC tuning and seems to
> endorse the G1 GC for the situation I'm working with. It appears to reduces
> long pauses at the expense of higher CPU load, which sounds like just the
> trade-off I want to make.
>
> But when I try to run the G1 GC, it causes the JVM to segfault, pretty
> soon after starting. I've asked around with a few others in #elasticsearch,
> and it seems others are having similar issues with G1--many I've spoken to
> have tried it but ditched it when it segfaulted too much.
>
> However, everyone seems to "know somebody" or know of someone who has had
> success with G1.
>
> So ... is G1 working for anyone? If you have G1 working and no segfault
> issues, can you tell me a little more about your environment? (what OS,
> JVM, ES version, JVM flags, app settings etc.) It doesn't make sense to me
> that it is segfaulting and dying for me and others, but working okay for
> some. There has to be a reason, _if it is_ working for some--I'm growing
> skeptical that G1GC is working for anyone until I hear otherwise 🙂
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
## -- mvh
Runar Myklebust
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@dezmodue](https://discuss.elastic.co/u/dezmodue)
#### Post date: [August 21, 2013, 7:03am UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/4 "2013-08-21T07:03:07Z")
Hi,
Thanks for sharing, would you mind sharing also:
Heap size
Total memory available
Number of nodes
Index size
New generation heap size
Shard size
Number of queries second/indexing
Number of indexes
How did you get to 30 for UseCMSInitiatingOccupancyOnly?
Thanks
S
On 21 Aug 2013 08:46, "Runar Myklebust" [runar.a.m@gmail.com](mailto:runar.a.m@gmail.com) wrote:
> We've had no luck with G1 because of the trove4j issue, tried on several
> installations on different platforms.
>
> Instead, this seems to be the best solution we have got so far using big
> heaps:
> -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=30
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:NewRatio=4
>
> On Tue, Aug 20, 2013 at 7:02 PM, Nathan Dial [nathan.dial@gmail.com](mailto:nathan.dial@gmail.com)
> wrote:
>
> > I'm trying to tune ES for predictable response times under an
> > update-heavy load. Our big challenge has been long GC pauses (sometimes
> > stopping the app for up to 30 seconds).
> >
> > This post (
> > [http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html))
> > by Jörg Prante has a lot of knowledgable advice on GC tuning and seems to
> > endorse the G1 GC for the situation I'm working with. It appears to reduces
> > long pauses at the expense of higher CPU load, which sounds like just the
> > trade-off I want to make.
> >
> > But when I try to run the G1 GC, it causes the JVM to segfault, pretty
> > soon after starting. I've asked around with a few others in #elasticsearch,
> > and it seems others are having similar issues with G1--many I've spoken to
> > have tried it but ditched it when it segfaulted too much.
> >
> > However, everyone seems to "know somebody" or know of someone who has
> > had success with G1.
> >
> > So ... is G1 working for anyone? If you have G1 working and no segfault
> > issues, can you tell me a little more about your environment? (what OS,
> > JVM, ES version, JVM flags, app settings etc.) It doesn't make sense to me
> > that it is segfaulting and dying for me and others, but working okay for
> > some. There has to be a reason, if it is working for some--I'm growing
> > skeptical that G1GC is working for anyone until I hear otherwise 🙂
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> ## -- mvh
>
> Runar Myklebust
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> G
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Runar\_Myklebust](https://discuss.elastic.co/u/Runar_Myklebust)
#### Post date: [August 21, 2013, 7:30am UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/5 "2013-08-21T07:30:12Z")
Hi, since ES is embedded in our product and used by different customers,
its hard to write any specific numbers on these, but generally this is what
we recommend when heap-size is, say, larger than 6GB. One customer has
heapsize of 20GB.
Number of nodes vary from 4 to 14. The index size is not very large in
number of documents, i guess around 266.000 is max I have seen, but there
are often a lot of data pr document (including extracted text from pdf's
etc)
The 30 for UCIOO was proved to work best at one of our customers after a
lot of testing with expected max traffic (200 request pr second, each
spawning from 4-20 queries on index) (in our product the central DB will
always be the the bottleneck of performance, not ES 🙂
On Wed, Aug 21, 2013 at 9:03 AM, Simone Sciarrati [s.sciarrati@gmail.com](mailto:s.sciarrati@gmail.com)wrote:
> Hi,
>
> Thanks for sharing, would you mind sharing also:
>
> Heap size
> Total memory available
> Number of nodes
> Index size
> New generation heap size
> Shard size
> Number of queries second/indexing
> Number of indexes
>
> How did you get to 30 for UseCMSInitiatingOccupancyOnly?
>
> Thanks
>
> S
>
> On 21 Aug 2013 08:46, "Runar Myklebust" [runar.a.m@gmail.com](mailto:runar.a.m@gmail.com) wrote:
>
> > We've had no luck with G1 because of the trove4j issue, tried on several
> > installations on different platforms.
> >
> > Instead, this seems to be the best solution we have got so far using big
> > heaps:
> > -XX:+UseConcMarkSweepGC
> > -XX:CMSInitiatingOccupancyFraction=30
> > -XX:+UseCMSInitiatingOccupancyOnly
> > -XX:NewRatio=4
> >
> > On Tue, Aug 20, 2013 at 7:02 PM, Nathan Dial [nathan.dial@gmail.com](mailto:nathan.dial@gmail.com)
> > wrote:
> >
> > > I'm trying to tune ES for predictable response times under an
> > > update-heavy load. Our big challenge has been long GC pauses (sometimes
> > > stopping the app for up to 30 seconds).
> > >
> > > This post (
> > > [http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html))
> > > by Jörg Prante has a lot of knowledgable advice on GC tuning and seems to
> > > endorse the G1 GC for the situation I'm working with. It appears to reduces
> > > long pauses at the expense of higher CPU load, which sounds like just the
> > > trade-off I want to make.
> > >
> > > But when I try to run the G1 GC, it causes the JVM to segfault, pretty
> > > soon after starting. I've asked around with a few others in #elasticsearch,
> > > and it seems others are having similar issues with G1--many I've spoken to
> > > have tried it but ditched it when it segfaulted too much.
> > >
> > > However, everyone seems to "know somebody" or know of someone who has
> > > had success with G1.
> > >
> > > So ... is G1 working for anyone? If you have G1 working and no segfault
> > > issues, can you tell me a little more about your environment? (what OS,
> > > JVM, ES version, JVM flags, app settings etc.) It doesn't make sense to me
> > > that it is segfaulting and dying for me and others, but working okay for
> > > some. There has to be a reason, if it is working for some--I'm growing
> > > skeptical that G1GC is working for anyone until I hear otherwise 🙂
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > ## -- mvh
> >
> > Runar Myklebust
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > G
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
## -- mvh
Runar Myklebust
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ndial](https://discuss.elastic.co/u/ndial)
#### Post date: [August 21, 2013, 1:57pm UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/6 "2013-08-21T13:57:39Z")
Thanks for the info. Does your NewRatio of 4 work for the 6 GB heaps as
well as the 30GB, or do you tend to have further tweaks in there?
What is your strategy for tweaking? do you use metrics to give you a
direction to move things, or is it all just trial and error?
On Wednesday, August 21, 2013 2:30:12 AM UTC-5, Runar Myklebust wrote:
> Hi, since ES is embedded in our product and used by different customers,
> its hard to write any specific numbers on these, but generally this is what
> we recommend when heap-size is, say, larger than 6GB. One customer has
> heapsize of 20GB.
> Number of nodes vary from 4 to 14. The index size is not very large in
> number of documents, i guess around 266.000 is max I have seen, but there
> are often a lot of data pr document (including extracted text from pdf's
> etc)
>
> The 30 for UCIOO was proved to work best at one of our customers after a
> lot of testing with expected max traffic (200 request pr second, each
> spawning from 4-20 queries on index) (in our product the central DB will
> always be the the bottleneck of performance, not ES 🙂
>
> On Wed, Aug 21, 2013 at 9:03 AM, Simone Sciarrati \<[s.sci...@gmail.com](mailto:s.sci...@gmail.com)\
>
> > wrote:
>
> > Hi,
> >
> > Thanks for sharing, would you mind sharing also:
> >
> > Heap size
> > Total memory available
> > Number of nodes
> > Index size
> > New generation heap size
> > Shard size
> > Number of queries second/indexing
> > Number of indexes
> >
> > How did you get to 30 for UseCMSInitiatingOccupancyOnly?
> >
> > Thanks
> >
> > S
> >
> > On 21 Aug 2013 08:46, "Runar Myklebust" \<[runa...@gmail.com](mailto:runa...@gmail.com) \\>
> > wrote:
> >
> > > We've had no luck with G1 because of the trove4j issue, tried on
> > > several installations on different platforms.
> > >
> > > Instead, this seems to be the best solution we have got so far using
> > > big heaps:
> > > -XX:+UseConcMarkSweepGC
> > > -XX:CMSInitiatingOccupancyFraction=30
> > > -XX:+UseCMSInitiatingOccupancyOnly
> > > -XX:NewRatio=4
> > >
> > > On Tue, Aug 20, 2013 at 7:02 PM, Nathan Dial \<[natha...@gmail.com](mailto:natha...@gmail.com)\\>
> > > wrote:
> > >
> > > > I'm trying to tune ES for predictable response times under an
> > > > update-heavy load. Our big challenge has been long GC pauses (sometimes
> > > > stopping the app for up to 30 seconds).
> > > >
> > > > This post (
> > > > [http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html))
> > > > by Jörg Prante has a lot of knowledgable advice on GC tuning and seems to
> > > > endorse the G1 GC for the situation I'm working with. It appears to reduces
> > > > long pauses at the expense of higher CPU load, which sounds like just the
> > > > trade-off I want to make.
> > > >
> > > > But when I try to run the G1 GC, it causes the JVM to segfault, pretty
> > > > soon after starting. I've asked around with a few others in #elasticsearch,
> > > > and it seems others are having similar issues with G1--many I've spoken to
> > > > have tried it but ditched it when it segfaulted too much.
> > > >
> > > > However, everyone seems to "know somebody" or know of someone who has
> > > > had success with G1.
> > > >
> > > > So ... is G1 working for anyone? If you have G1 working and no
> > > > segfault issues, can you tell me a little more about your environment?
> > > > (what OS, JVM, ES version, JVM flags, app settings etc.) It doesn't make
> > > > sense to me that it is segfaulting and dying for me and others, but working
> > > > okay for some. There has to be a reason, if it is working for some--I'm
> > > > growing skeptical that G1GC is working for anyone until I hear otherwise 🙂
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > >
> > > ## -- mvh
> > >
> > > Runar Myklebust
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > G
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> ## -- mvh
>
> Runar Myklebust
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:20am UTC](https://discuss.elastic.co/t/anyone-have-g1-gc-working-what-environment-configs/13274/7 "2017-07-06T02:20:20Z")