# Lot of GC in elasticsearch node **URL:** \ **Category:** Elasticsearch\ **Created:** [May 12, 2014, 6:38am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454 "2014-05-12T06:38:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![erbdex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erbdex/32/1520_2.png) [@erbdex](https://discuss.elastic.co/u/erbdex)\ **Post date:** [May 12, 2014, 6:38am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/1 "2014-05-12T06:38:33Z")
My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). Elasticsearch starts as- 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 /usr/bin/java _-Xms4g -Xmx4g -Xss256k_ -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch The node stopped responding (the ip:9200 status page), and so did kibana. It started working fine on a restart. i have logstash format docs wherein the index rotates daily. Stats: Daily: ~11G docs, ~15 million. Total: 195G docs, ~300 million. The logs of the time when it stopped responding are- [2014-05-12 03:39:08,789][INFO][cluster.metadata] [Hannibal King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) [2014-05-12 03:40:52,293][INFO][monitor.jvm] [Hannibal King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [150.3mb]-\>[1.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 03:44:11,739][INFO][cluster.metadata] [Hannibal King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) [2014-05-12 03:45:32,191][INFO][monitor.jvm] [Hannibal King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], total [5.2s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [197.4mb]-\>[9.3mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.5gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:06:01,224][INFO][monitor.jvm] [Hannibal King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total [6s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [134.7mb]-\>[9.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} [2014-05-12 04:08:14,473][INFO][monitor.jvm] [Hannibal King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], total [5.8s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [165.1mb]-\>[2.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:09:07,473][INFO][monitor.jvm] [Hannibal King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], total [6.2s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [265.9mb]-\>[2.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} [2014-05-12 04:10:08,387][INFO][monitor.jvm] [Hannibal King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], total [5.4s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] [176.5mb]-\>[5.8mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} [2014-05-12 04:13:12,774][INFO][monitor.jvm] [Hannibal King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], total [5.6s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] [167.4mb]-\>[12.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} [2014-05-12 04:14:22,729][INFO][monitor.jvm] [Hannibal King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [224.2mb]-\>[3.5mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:15:12,192][INFO][monitor.jvm] [Hannibal King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], total [5.2s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [234mb]-\>[2.4mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:15:32,344][INFO][monitor.jvm] [Hannibal King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], total [6.3s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [1.2mb]-\>[34.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:15:39,627][INFO][monitor.jvm] [Hannibal King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], total [6.7s]/[4.7h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [34.7mb]-\>[45.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:15:51,547][INFO][monitor.jvm] [Hannibal King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total [6.5s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [24.4mb]-\>[32.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:03,300][INFO][monitor.jvm] [Hannibal King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], total [5.8s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] [266.2mb]-\>[21.9mb]/[266.2mb]}{[survivor] [23.4mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:12,454][INFO][monitor.jvm] [Hannibal King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total [6.6s]/[4.7h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [196.8mb]-\>[33.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:24,635][INFO][monitor.jvm] [Hannibal King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], total [5.7s]/[4.8h], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [246.4mb]-\>[44.3mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:32,239][INFO][monitor.jvm] [Hannibal King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], total [6.4s]/[4.8h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [221.5mb]-\>[31.9mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:43,990][INFO][monitor.jvm] [Hannibal King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], total [6.4s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [40.5mb]-\>[64.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:51,259][INFO][monitor.jvm] [Hannibal King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], total [6.6s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [64.2mb]-\>[74.6mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:16:58,791][INFO][monitor.jvm] [Hannibal King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total [7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [74.6mb]-\>[89.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} [2014-05-12 04:17:06,028][INFO][monitor.jvm] [Hannibal King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], total [6.7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] [89.7mb]-\>[93.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} : -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2014, 8:33am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/2 "2014-05-12T08:33:19Z")
You need to reduce your data size, add more memory or add another node. Basically, you've reached the limits of that node. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 12 May 2014 16:38, Abhishek Tiwari [erbdex@gmail.com](mailto:erbdex@gmail.com) wrote: > My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). > Elasticsearch starts as- > > 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 > /usr/bin/java _-Xms4g -Xmx4g -Xss256k_ -Djava.awt.headless=true > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch > -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid > -Des.path.home=/usr/share/elasticsearch -cp > :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > -Des.default.path.home=/usr/share/elasticsearch > -Des.default.path.logs=/var/log/elasticsearch > -Des.default.path.data=/var/lib/elasticsearch > -Des.default.path.work=/tmp/elasticsearch > -Des.default.path.conf=/etc/elasticsearch > org.elasticsearch.bootstrap.Elasticsearch > > The node stopped responding (the ip:9200 status page), and so did kibana. > It started working fine on a restart. > i have logstash format docs wherein the index rotates daily. > Stats: > Daily: ~11G docs, ~15 million. > Total: 195G docs, ~300 million. > > The logs of the time when it stopped responding are- > > [2014-05-12 03:39:08,789][INFO][cluster.metadata] [Hannibal > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > [2014-05-12 03:40:52,293][INFO][monitor.jvm] [Hannibal > King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [150.3mb]-\>[1.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 03:44:11,739][INFO][cluster.metadata] [Hannibal > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > [2014-05-12 03:45:32,191][INFO][monitor.jvm] [Hannibal > King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], > total [5.2s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [197.4mb]-\>[9.3mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.5gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:06:01,224][INFO][monitor.jvm] [Hannibal > King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total > [6s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [134.7mb]-\>[9.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.5gb]/[3.6gb]} > [2014-05-12 04:08:14,473][INFO][monitor.jvm] [Hannibal > King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], > total [5.8s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [165.1mb]-\>[2.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:09:07,473][INFO][monitor.jvm] [Hannibal > King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], > total [6.2s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [265.9mb]-\>[2.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.5gb]/[3.6gb]} > [2014-05-12 04:10:08,387][INFO][monitor.jvm] [Hannibal > King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], > total [5.4s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > [176.5mb]-\>[5.8mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.5gb]/[3.6gb]} > [2014-05-12 04:13:12,774][INFO][monitor.jvm] [Hannibal > King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], > total [5.6s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > [167.4mb]-\>[12.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.5gb]/[3.6gb]} > [2014-05-12 04:14:22,729][INFO][monitor.jvm] [Hannibal > King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [224.2mb]-\>[3.5mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:15:12,192][INFO][monitor.jvm] [Hannibal > King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], > total [5.2s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [234mb]-\>[2.4mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:15:32,344][INFO][monitor.jvm] [Hannibal > King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], > total [6.3s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [1.2mb]-\>[34.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:15:39,627][INFO][monitor.jvm] [Hannibal > King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], > total [6.7s]/[4.7h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [34.7mb]-\>[45.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:15:51,547][INFO][monitor.jvm] [Hannibal > King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total > [6.5s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [24.4mb]-\>[32.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:03,300][INFO][monitor.jvm] [Hannibal > King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], > total [5.8s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > [266.2mb]-\>[21.9mb]/[266.2mb]}{[survivor] [23.4mb]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:12,454][INFO][monitor.jvm] [Hannibal > King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total > [6.6s]/[4.7h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [196.8mb]-\>[33.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:24,635][INFO][monitor.jvm] [Hannibal > King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], > total [5.7s]/[4.8h], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [246.4mb]-\>[44.3mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:32,239][INFO][monitor.jvm] [Hannibal > King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], > total [6.4s]/[4.8h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [221.5mb]-\>[31.9mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:43,990][INFO][monitor.jvm] [Hannibal > King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], > total [6.4s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [40.5mb]-\>[64.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:51,259][INFO][monitor.jvm] [Hannibal > King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], > total [6.6s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [64.2mb]-\>[74.6mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:16:58,791][INFO][monitor.jvm] [Hannibal > King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total > [7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [74.6mb]-\>[89.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > [2014-05-12 04:17:06,028][INFO][monitor.jvm] [Hannibal > King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], > total [6.7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > [89.7mb]-\>[93.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > [3.6gb]-\>[3.6gb]/[3.6gb]} > : > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZnCb6Ew2MggBPFuNpFr\_gHo\_vh6cUHgm5Nmp6X6G7uQQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZnCb6Ew2MggBPFuNpFr_gHo_vh6cUHgm5Nmp6X6G7uQQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![erbdex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erbdex/32/1520_2.png) [@erbdex](https://discuss.elastic.co/u/erbdex)\ **Post date:** [May 12, 2014, 9:33am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/3 "2014-05-12T09:33:45Z")
> add more memory i am doing 15 million docs, which total to ~9G. The average doc size is ~2KB. 1. How much memory would you suggest for my use-case? 2.Also, is it prudent for me to have half of OS memory dedicated to elasticsearch? On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote: > You need to reduce your data size, add more memory or add another node. > > Basically, you've reached the limits of that node. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 12 May 2014 16:38, Abhishek Tiwari \<[erb...@gmail.com](mailto:erb...@gmail.com) \\>wrote: > > > My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). > > Elasticsearch starts as- > > > > 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 > > /usr/bin/java _-Xms4g -Xmx4g -Xss256k_ -Djava.awt.headless=true > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > > -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch > > -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid > > -Des.path.home=/usr/share/elasticsearch -cp > > :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > > -Des.default.path.home=/usr/share/elasticsearch > > -Des.default.path.logs=/var/log/elasticsearch > > -Des.default.path.data=/var/lib/elasticsearch > > -Des.default.path.work=/tmp/elasticsearch > > -Des.default.path.conf=/etc/elasticsearch > > org.elasticsearch.bootstrap.Elasticsearch > > > > The node stopped responding (the ip:9200 status page), and so did kibana. > > It started working fine on a restart. > > i have logstash format docs wherein the index rotates daily. > > Stats: > > Daily: ~11G docs, ~15 million. > > Total: 195G docs, ~300 million. > > > > The logs of the time when it stopped responding are- > > > > [2014-05-12 03:39:08,789][INFO][cluster.metadata] [Hannibal > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > [2014-05-12 03:40:52,293][INFO][monitor.jvm] [Hannibal > > King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [150.3mb]-\>[1.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 03:44:11,739][INFO][cluster.metadata] [Hannibal > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > [2014-05-12 03:45:32,191][INFO][monitor.jvm] [Hannibal > > King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], > > total [5.2s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [197.4mb]-\>[9.3mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.5gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:06:01,224][INFO][monitor.jvm] [Hannibal > > King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total > > [6s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [134.7mb]-\>[9.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > [2014-05-12 04:08:14,473][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], > > total [5.8s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [165.1mb]-\>[2.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:09:07,473][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], > > total [6.2s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [265.9mb]-\>[2.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > [2014-05-12 04:10:08,387][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], > > total [5.4s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > [176.5mb]-\>[5.8mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > [2014-05-12 04:13:12,774][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], > > total [5.6s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > [167.4mb]-\>[12.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > [2014-05-12 04:14:22,729][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [224.2mb]-\>[3.5mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:15:12,192][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], > > total [5.2s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [234mb]-\>[2.4mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:15:32,344][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], > > total [6.3s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [1.2mb]-\>[34.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:15:39,627][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], > > total [6.7s]/[4.7h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [34.7mb]-\>[45.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:15:51,547][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total > > [6.5s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [24.4mb]-\>[32.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:03,300][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], > > total [5.8s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > [266.2mb]-\>[21.9mb]/[266.2mb]}{[survivor] [23.4mb]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:12,454][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total > > [6.6s]/[4.7h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [196.8mb]-\>[33.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:24,635][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], > > total [5.7s]/[4.8h], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [246.4mb]-\>[44.3mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:32,239][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], > > total [6.4s]/[4.8h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [221.5mb]-\>[31.9mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:43,990][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], > > total [6.4s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [40.5mb]-\>[64.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:51,259][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], > > total [6.6s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [64.2mb]-\>[74.6mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:16:58,791][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total > > [7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [74.6mb]-\>[89.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > [2014-05-12 04:17:06,028][INFO][monitor.jvm] [Hannibal > > King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], > > total [6.7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > [89.7mb]-\>[93.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > : > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2014, 10:00am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/4 "2014-05-12T10:00:36Z")
It's standard practise to use 50% of system memory for the heap. How much RAM you need depends on how long you want to keep your data around for. So, given you have ~200GB now on 4GB of RAM, you can probably extrapolate that out based on your needs. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 12 May 2014 19:33, Abhishek Tiwari [erbdex@gmail.com](mailto:erbdex@gmail.com) wrote: > add more memory > > i am doing 15 million docs, which total to ~9G. The average doc size is > ~2KB. > > 1. How much memory would you suggest for my use-case? > 2.Also, is it prudent for me to have half of OS memory dedicated to > elasticsearch? > > On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote: > > > You need to reduce your data size, add more memory or add another node. > > > > Basically, you've reached the limits of that node. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 12 May 2014 16:38, Abhishek Tiwari [erb...@gmail.com](mailto:erb...@gmail.com) wrote: > > > > > My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). > > > Elasticsearch starts as- > > > > > > 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 > > > /usr/bin/java _-Xms4g -Xmx4g -Xss256k_ -Djava.awt.headless=true > > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > > > CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > > > -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/ > > > elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch > > > -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/ > > > share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > > > -Des.default.path.home=/usr/share/elasticsearch > > > -Des.default.path.logs=/var/log/elasticsearch > > > -Des.default.path.data=/var/lib/elasticsearch > > > -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch > > > org.elasticsearch.bootstrap.Elasticsearch > > > > > > The node stopped responding (the ip:9200 status page), and so did > > > kibana. It started working fine on a restart. > > > i have logstash format docs wherein the index rotates daily. > > > Stats: > > > Daily: ~11G docs, ~15 million. > > > Total: 195G docs, ~300 million. > > > > > > The logs of the time when it stopped responding are- > > > > > > [2014-05-12 03:39:08,789][INFO][cluster.metadata] [Hannibal > > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > > [2014-05-12 03:40:52,293][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], > > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [150.3mb]-\>[1.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 03:44:11,739][INFO][cluster.metadata] [Hannibal > > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > > [2014-05-12 03:45:32,191][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], > > > total [5.2s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [197.4mb]-\>[9.3mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.5gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:06:01,224][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total > > > [6s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [134.7mb]-\>[9.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > > [2014-05-12 04:08:14,473][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], > > > total [5.8s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [165.1mb]-\>[2.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:09:07,473][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], > > > total [6.2s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [265.9mb]-\>[2.9mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > > [2014-05-12 04:10:08,387][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], > > > total [5.4s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > > [176.5mb]-\>[5.8mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.5gb]/[3.6gb]} > > > [2014-05-12 04:13:12,774][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], > > > total [5.6s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > > [167.4mb]-\>[12.9mb]/[266.2mb]}{[survivor] > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > [2014-05-12 04:14:22,729][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], > > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [224.2mb]-\>[3.5mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:15:12,192][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], > > > total [5.2s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [234mb]-\>[2.4mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:15:32,344][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], > > > total [6.3s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [1.2mb]-\>[34.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:15:39,627][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], > > > total [6.7s]/[4.7h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [34.7mb]-\>[45.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:15:51,547][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total > > > [6.5s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [24.4mb]-\>[32.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:03,300][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], > > > total [5.8s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > [266.2mb]-\>[21.9mb]/[266.2mb]}{[survivor] > > > [23.4mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:12,454][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total > > > [6.6s]/[4.7h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [196.8mb]-\>[33.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:24,635][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], > > > total [5.7s]/[4.8h], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [246.4mb]-\>[44.3mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:32,239][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], > > > total [6.4s]/[4.8h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [221.5mb]-\>[31.9mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:43,990][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], > > > total [6.4s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [40.5mb]-\>[64.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:51,259][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], > > > total [6.6s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [64.2mb]-\>[74.6mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:16:58,791][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total > > > [7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [74.6mb]-\>[89.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > [2014-05-12 04:17:06,028][INFO][monitor.jvm] [Hannibal > > > King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], > > > total [6.7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > [89.7mb]-\>[93.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > : > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92% > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624agdKkWdFCXtbEW%3DH4QJg8edsAuMZqF9ZpBZ%3DJeZew2BA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624agdKkWdFCXtbEW%3DH4QJg8edsAuMZqF9ZpBZ%3DJeZew2BA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![erbdex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erbdex/32/1520_2.png) [@erbdex](https://discuss.elastic.co/u/erbdex)\ **Post date:** [May 12, 2014, 10:45am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/5 "2014-05-12T10:45:20Z")
> How much RAM you need depends on how long you want to keep your data > around for. So, given you have ~200GB now on 4GB of RAM, you can probably > extrapolate that out based on your needs. Isn't my problem more with 9G _daily_ index, than with total of 200G(20 days x 9G) indexes? Correct me if i am wrong here but doesn't kibana ask elasticsearch for just one day/week of indices(based on the query). Will elasticsearch really care if i have 500 days of total day-wise segregated indices out there but am performing queries on _just past 7 days_? Is this a total-footprint problem or a daily throughput problem? On Monday, 12 May 2014 15:30:36 UTC+5:30, Mark Walkom wrote: > It's standard practise to use 50% of system memory for the heap. > > How much RAM you need depends on how long you want to keep your data > around for. So, given you have ~200GB now on 4GB of RAM, you can probably > extrapolate that out based on your needs. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 12 May 2014 19:33, Abhishek Tiwari \<[erb...@gmail.com](mailto:erb...@gmail.com) \\>wrote: > > > add more memory > > > > i am doing 15 million docs, which total to ~9G. The average doc size is > > ~2KB. > > > > 1. How much memory would you suggest for my use-case? > > 2.Also, is it prudent for me to have half of OS memory dedicated to > > elasticsearch? > > > > On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote: > > > > > You need to reduce your data size, add more memory or add another node. > > > > > > Basically, you've reached the limits of that node. > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 12 May 2014 16:38, Abhishek Tiwari [erb...@gmail.com](mailto:erb...@gmail.com) wrote: > > > > > > > My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). > > > > Elasticsearch starts as- > > > > > > > > 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 > > > > /usr/bin/java _-Xms4g -Xmx4g -Xss256k_ -Djava.awt.headless=true > > > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > > > > CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > > > > -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/ > > > > elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch > > > > -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/ > > > > share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > > > > -Des.default.path.home=/usr/share/elasticsearch > > > > -Des.default.path.logs=/var/log/elasticsearch > > > > -Des.default.path.data=/var/lib/elasticsearch > > > > -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch > > > > org.elasticsearch.bootstrap.Elasticsearch > > > > > > > > The node stopped responding (the ip:9200 status page), and so did > > > > kibana. It started working fine on a restart. > > > > i have logstash format docs wherein the index rotates daily. > > > > Stats: > > > > Daily: ~11G docs, ~15 million. > > > > Total: 195G docs, ~300 million. > > > > > > > > The logs of the time when it stopped responding are- > > > > > > > > [2014-05-12 03:39:08,789][INFO][cluster.metadata] [Hannibal > > > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > > > [2014-05-12 03:40:52,293][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], > > > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [150.3mb]-\>[1.7mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 03:44:11,739][INFO][cluster.metadata] [Hannibal > > > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > > > [2014-05-12 03:45:32,191][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], > > > > total [5.2s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [197.4mb]-\>[9.3mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.5gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:06:01,224][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total > > > > [6s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [134.7mb]-\>[9.9mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > [2014-05-12 04:08:14,473][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], > > > > total [5.8s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [165.1mb]-\>[2.7mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:09:07,473][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], > > > > total [6.2s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [265.9mb]-\>[2.9mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > [2014-05-12 04:10:08,387][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], > > > > total [5.4s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > > > [176.5mb]-\>[5.8mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > [2014-05-12 04:13:12,774][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], > > > > total [5.6s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > > > [167.4mb]-\>[12.9mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > [2014-05-12 04:14:22,729][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], > > > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [224.2mb]-\>[3.5mb]/[266.2mb]}{[survivor] > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:15:12,192][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], > > > > total [5.2s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [234mb]-\>[2.4mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:15:32,344][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], > > > > total [6.3s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [1.2mb]-\>[34.7mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:15:39,627][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], > > > > total [6.7s]/[4.7h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [34.7mb]-\>[45.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:15:51,547][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total > > > > [6.5s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [24.4mb]-\>[32.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:03,300][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], > > > > total [5.8s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > [266.2mb]-\>[21.9mb]/[266.2mb]}{[survivor] > > > > [23.4mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:12,454][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total > > > > [6.6s]/[4.7h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [196.8mb]-\>[33.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:24,635][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], > > > > total [5.7s]/[4.8h], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [246.4mb]-\>[44.3mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:32,239][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], > > > > total [6.4s]/[4.8h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [221.5mb]-\>[31.9mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:43,990][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], > > > > total [6.4s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [40.5mb]-\>[64.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:51,259][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], > > > > total [6.6s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [64.2mb]-\>[74.6mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:16:58,791][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total > > > > [7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [74.6mb]-\>[89.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > [2014-05-12 04:17:06,028][INFO][monitor.jvm] [Hannibal > > > > King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], > > > > total [6.7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > [89.7mb]-\>[93.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > : > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 12, 2014, 10:47am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/6 "2014-05-12T10:47:11Z")
Yes, kibana will load for whatever you ask for, _but_ ES has to maintain index metadata for every index in memory. Those two coupled are pushing things too far for your heap. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 12 May 2014 20:45, Abhishek Tiwari [erbdex@gmail.com](mailto:erbdex@gmail.com) wrote: > How much RAM you need depends on how long you want to keep your data > > > around for. So, given you have ~200GB now on 4GB of RAM, you can probably > > extrapolate that out based on your needs. > > Isn't my problem more with 9G _daily_ index, than with total of 200G(20 > days x 9G) indexes? > Correct me if i am wrong here but doesn't kibana ask elasticsearch for > just one day/week of indices(based on the query). > Will elasticsearch really care if i have 500 days of total day-wise > segregated indices out there but am performing queries on _just past 7 > days_? > Is this a total-footprint problem or a daily throughput problem? > > On Monday, 12 May 2014 15:30:36 UTC+5:30, Mark Walkom wrote: > > > It's standard practise to use 50% of system memory for the heap. > > > > How much RAM you need depends on how long you want to keep your data > > around for. So, given you have ~200GB now on 4GB of RAM, you can probably > > extrapolate that out based on your needs. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 12 May 2014 19:33, Abhishek Tiwari [erb...@gmail.com](mailto:erb...@gmail.com) wrote: > > > > > add more memory > > > > > > i am doing 15 million docs, which total to ~9G. The average doc size is > > > ~2KB. > > > > > > 1. How much memory would you suggest for my use-case? > > > 2.Also, is it prudent for me to have half of OS memory dedicated to > > > elasticsearch? > > > > > > On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote: > > > > > > > You need to reduce your data size, add more memory or add another node. > > > > > > > > Basically, you've reached the limits of that node. > > > > > > > > Regards, > > > > Mark Walkom > > > > > > > > Infrastructure Engineer > > > > Campaign Monitor > > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > > > On 12 May 2014 16:38, Abhishek Tiwari [erb...@gmail.com](mailto:erb...@gmail.com) wrote: > > > > > > > > > My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem). > > > > > Elasticsearch starts as- > > > > > > > > > > 498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10 > > > > > /usr/bin/java _-Xms4g -Xmx4g -Xss256k_ -Djava.awt.headless=true > > > > > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > > > > > CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > > > > > -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/ > > > > > elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch > > > > > -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/s > > > > > hare/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > > > > > -Des.default.path.home=/usr/share/elasticsearch > > > > > -Des.default.path.logs=/var/log/elasticsearch > > > > > -Des.default.path.data=/var/lib/elasticsearch > > > > > -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/ > > > > > elasticsearch org.elasticsearch.bootstrap.Elasticsearch > > > > > > > > > > The node stopped responding (the ip:9200 status page), and so did > > > > > kibana. It started working fine on a restart. > > > > > i have logstash format docs wherein the index rotates daily. > > > > > Stats: > > > > > Daily: ~11G docs, ~15 million. > > > > > Total: 195G docs, ~300 million. > > > > > > > > > > The logs of the time when it stopped responding are- > > > > > > > > > > [2014-05-12 03:39:08,789][INFO][cluster.metadata] [Hannibal > > > > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > > > > [2014-05-12 03:40:52,293][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], > > > > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [150.3mb]-\>[1.7mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 03:44:11,739][INFO][cluster.metadata] [Hannibal > > > > > King] [logstash-2014.05.12] update\_mapping [medusa\_ex] (dynamic) > > > > > [2014-05-12 03:45:32,191][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], > > > > > total [5.2s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [197.4mb]-\>[9.3mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.5gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:06:01,224][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total > > > > > [6s]/[4.7h], memory [3.7gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [134.7mb]-\>[9.9mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > > [2014-05-12 04:08:14,473][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], > > > > > total [5.8s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [165.1mb]-\>[2.7mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:09:07,473][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], > > > > > total [6.2s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [265.9mb]-\>[2.9mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > > [2014-05-12 04:10:08,387][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], > > > > > total [5.4s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > > > > [176.5mb]-\>[5.8mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > > [2014-05-12 04:13:12,774][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], > > > > > total [5.6s]/[4.7h], memory [3.8gb]-\>[3.5gb]/[3.9gb], all\_pools {[young] > > > > > [167.4mb]-\>[12.9mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.5gb]/[3.6gb]} > > > > > [2014-05-12 04:14:22,729][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], > > > > > total [6.3s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [224.2mb]-\>[3.5mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:15:12,192][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], > > > > > total [5.2s]/[4.7h], memory [3.8gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [234mb]-\>[2.4mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:15:32,344][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], > > > > > total [6.3s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [1.2mb]-\>[34.7mb]/[266.2mb]}{[survivor] > > > > > [33.2mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:15:39,627][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], > > > > > total [6.7s]/[4.7h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [34.7mb]-\>[45.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:15:51,547][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total > > > > > [6.5s]/[4.7h], memory [3.6gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [24.4mb]-\>[32.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:03,300][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], > > > > > total [5.8s]/[4.7h], memory [3.9gb]-\>[3.6gb]/[3.9gb], all\_pools {[young] > > > > > [266.2mb]-\>[21.9mb]/[266.2mb]}{[survivor] > > > > > [23.4mb]-\>[0b]/[33.2mb]}{[old] [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:12,454][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total > > > > > [6.6s]/[4.7h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [196.8mb]-\>[33.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:24,635][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], > > > > > total [5.7s]/[4.8h], memory [3.9gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [246.4mb]-\>[44.3mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:32,239][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], > > > > > total [6.4s]/[4.8h], memory [3.8gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [221.5mb]-\>[31.9mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:43,990][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], > > > > > total [6.4s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [40.5mb]-\>[64.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:51,259][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], > > > > > total [6.6s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [64.2mb]-\>[74.6mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:16:58,791][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total > > > > > [7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [74.6mb]-\>[89.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > [2014-05-12 04:17:06,028][INFO][monitor.jvm] [Hannibal > > > > > King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], > > > > > total [6.7s]/[4.8h], memory [3.7gb]-\>[3.7gb]/[3.9gb], all\_pools {[young] > > > > > [89.7mb]-\>[93.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] > > > > > [3.6gb]-\>[3.6gb]/[3.6gb]} > > > > > : > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40goo > > > > > [glegroups.com](http://glegroups.com)[https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bT-DR0oYaAcHOxgR0MM7SdoEM%3DXw\_GcRD9aGZT4uT4gA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bT-DR0oYaAcHOxgR0MM7SdoEM%3DXw_GcRD9aGZT4uT4gA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:30am UTC](https://discuss.elastic.co/t/lot-of-gc-in-elasticsearch-node/17454/7 "2017-07-06T01:30:10Z")