# Growing old-gen size **URL:** \ **Category:** Elasticsearch\ **Created:** [March 20, 2015, 8:25pm UTC](https://discuss.elastic.co/t/growing-old-gen-size/22785 "2015-03-20T20:25:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mjdude5](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mjdude5](https://discuss.elastic.co/u/mjdude5)\ **Post date:** [March 20, 2015, 8:25pm UTC](https://discuss.elastic.co/t/growing-old-gen-size/22785/1 "2015-03-20T20:25:55Z")
Hello, I'm trying to better understand our ES memory usage with relation to our workload. Right now 2 of our nodes have above-average heap usage. Looking at \_stats their old-gen is large, ~6gb. Here's the \_cat usage I've been trying to make sense of: \_cat/nodes?v&h=host,v,j,hm,fm,fcm,sm,siwm,svmm,sc,pm,im,fce,fe,hp host v j hm fm fcm sm siwm svmm sc pm im fce fe hp host1 1.3.4 1.7.0\_71 7.9gb 1.2gb 34.3mb 2.8gb 1.3mb 7.4kb 13144 -1b 0b 0 0 82 host2 1.3.4 1.7.0\_71 7.9gb 888.2mb 20.3mb 1.9gb 0b 0b 8962 -1b 0b 0 0 67 host3 1.3.4 1.7.0\_71 7.9gb 1.1gb 29mb 2.5gb 0b 0b 11070 -1b 0b 0 0 70 host4 1.3.4 1.7.0\_71 7.9gb 845.2mb 21.6mb 1.8gb 179.8kb 448b 8024 -1b 0b 0 0 55 host5 1.3.4 1.7.0\_71 7.9gb 1.3gb 40.7mb 2.8gb 0b 0b 12615 -1b 0b 0 0 83 host1 and host5 when they do GC it looks like they drop ~5-10% so they bump against the 75% mark again very soon afterwords. Their oldgen stays relatively big, host5 currently has ~6gb of old gen. Last week we had an incident where a node started having long GC times and then eventually dropped out of the cluster, so that's the fear. It didn't seem like the GC was making any progress, wasn't actually reducing the memory size. There must be something using heap that's not reflected in this \_cat output? The collection\_count for the oldgen is increasing but the used\_in\_bytes isn't significantly reducing. Is that expected? thanks for any tips! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 22, 2015, 10:42pm UTC](https://discuss.elastic.co/t/growing-old-gen-size/22785/2 "2015-03-22T22:42:02Z")
Sounds like you're just utilising your cluster to it's capacity. If you are seeing GC causing nodes to drop out you probably want to consider either moving to doc values, reducing your dataset or adding more nodes/heap. On 21 March 2015 at 07:25, [mjdude5@gmail.com](mailto:mjdude5@gmail.com) wrote: > Hello, I'm trying to better understand our ES memory usage with relation > to our workload. Right now 2 of our nodes have above-average heap usage. > Looking at \_stats their old-gen is large, ~6gb. Here's the \_cat usage I've > been trying to make sense of: > > \_cat/nodes?v&h=host,v,j,hm,fm,fcm,sm,siwm,svmm,sc,pm,im,fce,fe,hp > > host v j hm fm > fcm sm siwm svmm sc pm im fce fe hp > host1 1.3.4 1.7.0\_71 7.9gb 1.2gb 34.3mb > 2.8gb 1.3mb 7.4kb 13144 -1b 0b 0 0 82 > host2 1.3.4 1.7.0\_71 7.9gb 888.2mb 20.3mb > 1.9gb 0b 0b 8962 -1b 0b 0 0 67 > host3 1.3.4 1.7.0\_71 7.9gb 1.1gb 29mb > 2.5gb 0b 0b 11070 -1b 0b 0 0 70 > host4 1.3.4 1.7.0\_71 7.9gb 845.2mb 21.6mb > 1.8gb 179.8kb 448b 8024 -1b 0b 0 0 55 > host5 1.3.4 1.7.0\_71 7.9gb 1.3gb 40.7mb > 2.8gb 0b 0b 12615 -1b 0b 0 0 83 > > host1 and host5 when they do GC it looks like they drop ~5-10% so they > bump against the 75% mark again very soon afterwords. Their oldgen stays > relatively big, host5 currently has ~6gb of old gen. > > Last week we had an incident where a node started having long GC times and > then eventually dropped out of the cluster, so that's the fear. It didn't > seem like the GC was making any progress, wasn't actually reducing the > memory size. > > There must be something using heap that's not reflected in this \_cat > output? The collection\_count for the oldgen is increasing but the > used\_in\_bytes isn't significantly reducing. Is that expected? > > thanks for any tips! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_pDg%3DtnwgthPAJYvO7dEJce93ogkzFaxwbTWZioPn8BA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_pDg%3DtnwgthPAJYvO7dEJce93ogkzFaxwbTWZioPn8BA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mjdude5](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mjdude5](https://discuss.elastic.co/u/mjdude5)\ **Post date:** [March 23, 2015, 3:52pm UTC](https://discuss.elastic.co/t/growing-old-gen-size/22785/3 "2015-03-23T15:52:52Z")
Thanks, do you know if there's more memory metric reporting that I'm missing? I'd like to figure out what's growing the fastest/largest. Fielddata I think should show up in the 'fm' column of the node stats. I'm mostly curious what I'm missing in adding up the memory requirements, from the node stats I have so far fielddata and segment memory are the two dominant components but they don't add up to more than 50% of the max heap. I know there are some things missing from the stats like metadata memory usage, but I'm assuming those are smaller components. On Sunday, March 22, 2015 at 6:42:29 PM UTC-4, Mark Walkom wrote: > Sounds like you're just utilising your cluster to it's capacity. > > If you are seeing GC causing nodes to drop out you probably want to > consider either moving to doc values, reducing your dataset or adding more > nodes/heap. > > On 21 March 2015 at 07:25, \<[mjd...@gmail.com](mailto:mjd...@gmail.com) \\> wrote: > > > Hello, I'm trying to better understand our ES memory usage with relation > > to our workload. Right now 2 of our nodes have above-average heap usage. > > Looking at \_stats their old-gen is large, ~6gb. Here's the \_cat usage I've > > been trying to make sense of: > > > > \_cat/nodes?v&h=host,v,j,hm,fm,fcm,sm,siwm,svmm,sc,pm,im,fce,fe,hp > > > > host v j hm fm > > fcm sm siwm svmm sc pm im fce fe hp > > host1 1.3.4 1.7.0\_71 7.9gb 1.2gb 34.3mb > > 2.8gb 1.3mb 7.4kb 13144 -1b 0b 0 0 82 > > host2 1.3.4 1.7.0\_71 7.9gb 888.2mb 20.3mb > > 1.9gb 0b 0b 8962 -1b 0b 0 0 67 > > host3 1.3.4 1.7.0\_71 7.9gb 1.1gb 29mb > > 2.5gb 0b 0b 11070 -1b 0b 0 0 70 > > host4 1.3.4 1.7.0\_71 7.9gb 845.2mb 21.6mb > > 1.8gb 179.8kb 448b 8024 -1b 0b 0 0 55 > > host5 1.3.4 1.7.0\_71 7.9gb 1.3gb 40.7mb > > 2.8gb 0b 0b 12615 -1b 0b 0 0 83 > > > > host1 and host5 when they do GC it looks like they drop ~5-10% so they > > bump against the 75% mark again very soon afterwords. Their oldgen stays > > relatively big, host5 currently has ~6gb of old gen. > > > > Last week we had an incident where a node started having long GC times > > and then eventually dropped out of the cluster, so that's the fear. It > > didn't seem like the GC was making any progress, wasn't actually reducing > > the memory size. > > > > There must be something using heap that's not reflected in this \_cat > > output? The collection\_count for the oldgen is increasing but the > > used\_in\_bytes isn't significantly reducing. Is that expected? > > > > thanks for any tips! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5271138c-f4a6-4261-9c57-82bfcaec36fb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5271138c-f4a6-4261-9c57-82bfcaec36fb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 24, 2015, 12:03am UTC](https://discuss.elastic.co/t/growing-old-gen-size/22785/4 "2015-03-24T00:03:32Z")
There is a whole bunch more stats you can get from various APIs, see them listed here [Elastic: Search Results | Elastic](https://www.elastic.co/search?q=stats) On 24 March 2015 at 02:52, [mjdude5@gmail.com](mailto:mjdude5@gmail.com) wrote: > Thanks, do you know if there's more memory metric reporting that I'm > missing? I'd like to figure out what's growing the fastest/largest. > Fielddata I think should show up in the 'fm' column of the node stats. I'm > mostly curious what I'm missing in adding up the memory requirements, from > the node stats I have so far fielddata and segment memory are the two > dominant components but they don't add up to more than 50% of the max heap. > > I know there are some things missing from the stats like metadata memory > usage, but I'm assuming those are smaller components. > > On Sunday, March 22, 2015 at 6:42:29 PM UTC-4, Mark Walkom wrote: > > > Sounds like you're just utilising your cluster to it's capacity. > > > > If you are seeing GC causing nodes to drop out you probably want to > > consider either moving to doc values, reducing your dataset or adding more > > nodes/heap. > > > > On 21 March 2015 at 07:25, [mjd...@gmail.com](mailto:mjd...@gmail.com) wrote: > > > > > Hello, I'm trying to better understand our ES memory usage with relation > > > to our workload. Right now 2 of our nodes have above-average heap usage. > > > Looking at \_stats their old-gen is large, ~6gb. Here's the \_cat usage I've > > > been trying to make sense of: > > > > > > \_cat/nodes?v&h=host,v,j,hm,fm,fcm,sm,siwm,svmm,sc,pm,im,fce,fe,hp > > > > > > host v j hm fm > > > fcm sm siwm svmm sc pm im fce fe hp > > > host1 1.3.4 1.7.0\_71 7.9gb 1.2gb 34.3mb > > > 2.8gb 1.3mb 7.4kb 13144 -1b 0b 0 0 82 > > > host2 1.3.4 1.7.0\_71 7.9gb 888.2mb 20.3mb > > > 1.9gb 0b 0b 8962 -1b 0b 0 0 67 > > > host3 1.3.4 1.7.0\_71 7.9gb 1.1gb 29mb > > > 2.5gb 0b 0b 11070 -1b 0b 0 0 70 > > > host4 1.3.4 1.7.0\_71 7.9gb 845.2mb 21.6mb > > > 1.8gb 179.8kb 448b 8024 -1b 0b 0 0 55 > > > host5 1.3.4 1.7.0\_71 7.9gb 1.3gb 40.7mb > > > 2.8gb 0b 0b 12615 -1b 0b 0 0 83 > > > > > > host1 and host5 when they do GC it looks like they drop ~5-10% so they > > > bump against the 75% mark again very soon afterwords. Their oldgen stays > > > relatively big, host5 currently has ~6gb of old gen. > > > > > > Last week we had an incident where a node started having long GC times > > > and then eventually dropped out of the cluster, so that's the fear. It > > > didn't seem like the GC was making any progress, wasn't actually reducing > > > the memory size. > > > > > > There must be something using heap that's not reflected in this \_cat > > > output? The collection\_count for the oldgen is increasing but the > > > used\_in\_bytes isn't significantly reducing. Is that expected? > > > > > > thanks for any tips! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5812e26e-a04e-4e03-9990-0c84f843a4ef%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5271138c-f4a6-4261-9c57-82bfcaec36fb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5271138c-f4a6-4261-9c57-82bfcaec36fb%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/5271138c-f4a6-4261-9c57-82bfcaec36fb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5271138c-f4a6-4261-9c57-82bfcaec36fb%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_WN5uOqTHK2kJueP8Vwe6FyjdmLwiOz5%2BH2h-EeBryWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_WN5uOqTHK2kJueP8Vwe6FyjdmLwiOz5%2BH2h-EeBryWQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:24am UTC](https://discuss.elastic.co/t/growing-old-gen-size/22785/5 "2017-07-06T00:24:38Z")