Lot of GC in elasticsearch node


(erbdex) #1

My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem).
Elasticsearch starts as-

498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10
/usr/bin/java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
-Des.path.home=/usr/share/elasticsearch -cp
:/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

The node stopped responding (the ip:9200 status page), and so did kibana.
It started working fine on a restart.
i have logstash format docs wherein the index rotates daily.
Stats:
Daily: ~11G docs, ~15 million.
Total: 195G docs, ~300 million.

The logs of the time when it stopped responding are-

[2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal King]
[logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s], total
[6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[150.3mb]->[1.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal King]
[logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s], total
[5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[197.4mb]->[9.3mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.5gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total
[6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[134.7mb]->[9.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s], total
[5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[165.1mb]->[2.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s], total
[6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[265.9mb]->[2.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s], total
[5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[176.5mb]->[5.8mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s], total
[5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[167.4mb]->[12.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s], total
[6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[224.2mb]->[3.5mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s], total
[5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s], total
[6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[1.2mb]->[34.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s], total
[6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total
[6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s], total
[5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[266.2mb]->[21.9mb]/[266.2mb]}{[survivor] [23.4mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total
[6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s], total
[5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young]
[246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s], total
[6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s], total
[6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s], total
[6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total
[7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal King]
[gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s], total
[6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

You need to reduce your data size, add more memory or add another node.

Basically, you've reached the limits of that node.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 16:38, Abhishek Tiwari erbdex@gmail.com wrote:

My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem).
Elasticsearch starts as-

498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10
/usr/bin/java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
-Des.path.home=/usr/share/elasticsearch -cp
:/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

The node stopped responding (the ip:9200 status page), and so did kibana.
It started working fine on a restart.
i have logstash format docs wherein the index rotates daily.
Stats:
Daily: ~11G docs, ~15 million.
Total: 195G docs, ~300 million.

The logs of the time when it stopped responding are-

[2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[150.3mb]->[1.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s],
total [5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[197.4mb]->[9.3mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.5gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total
[6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[134.7mb]->[9.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[165.1mb]->[2.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s],
total [6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[265.9mb]->[2.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s],
total [5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[176.5mb]->[5.8mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s],
total [5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[167.4mb]->[12.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[224.2mb]->[3.5mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s],
total [5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s],
total [6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[1.2mb]->[34.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total
[6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[266.2mb]->[21.9mb]/[266.2mb]}{[survivor] [23.4mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total
[6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s],
total [5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young]
[246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s],
total [6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s],
total [6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s],
total [6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total
[7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
:

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZnCb6Ew2MggBPFuNpFr_gHo_vh6cUHgm5Nmp6X6G7uQQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(erbdex) #3

add more memory

i am doing 15 million docs, which total to ~9G. The average doc size is
~2KB.

  1. How much memory would you suggest for my use-case?
    2.Also, is it prudent for me to have half of OS memory dedicated to
    elasticsearch?

On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote:

You need to reduce your data size, add more memory or add another node.

Basically, you've reached the limits of that node.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 12 May 2014 16:38, Abhishek Tiwari <erb...@gmail.com <javascript:>>wrote:

My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem).
Elasticsearch starts as-

498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10
/usr/bin/java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
-Des.path.home=/usr/share/elasticsearch -cp
:/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

The node stopped responding (the ip:9200 status page), and so did kibana.
It started working fine on a restart.
i have logstash format docs wherein the index rotates daily.
Stats:
Daily: ~11G docs, ~15 million.
Total: 195G docs, ~300 million.

The logs of the time when it stopped responding are-

[2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[150.3mb]->[1.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s],
total [5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[197.4mb]->[9.3mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.5gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total
[6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[134.7mb]->[9.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[165.1mb]->[2.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s],
total [6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[265.9mb]->[2.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s],
total [5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[176.5mb]->[5.8mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s],
total [5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[167.4mb]->[12.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[224.2mb]->[3.5mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s],
total [5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s],
total [6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[1.2mb]->[34.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total
[6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[266.2mb]->[21.9mb]/[266.2mb]}{[survivor] [23.4mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total
[6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s],
total [5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young]
[246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s],
total [6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s],
total [6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s],
total [6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total
[7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
:

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

It's standard practise to use 50% of system memory for the heap.

How much RAM you need depends on how long you want to keep your data around
for. So, given you have ~200GB now on 4GB of RAM, you can probably
extrapolate that out based on your needs.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 19:33, Abhishek Tiwari erbdex@gmail.com wrote:

add more memory

i am doing 15 million docs, which total to ~9G. The average doc size is
~2KB.

  1. How much memory would you suggest for my use-case?
    2.Also, is it prudent for me to have half of OS memory dedicated to
    elasticsearch?

On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote:

You need to reduce your data size, add more memory or add another node.

Basically, you've reached the limits of that node.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 16:38, Abhishek Tiwari erb...@gmail.com wrote:

My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem).
Elasticsearch starts as-

498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10
/usr/bin/java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/
elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch
-cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/
share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

The node stopped responding (the ip:9200 status page), and so did
kibana. It started working fine on a restart.
i have logstash format docs wherein the index rotates daily.
Stats:
Daily: ~11G docs, ~15 million.
Total: 195G docs, ~300 million.

The logs of the time when it stopped responding are-

[2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[150.3mb]->[1.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s],
total [5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[197.4mb]->[9.3mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.5gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total
[6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[134.7mb]->[9.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[165.1mb]->[2.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s],
total [6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[265.9mb]->[2.9mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s],
total [5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[176.5mb]->[5.8mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s],
total [5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[167.4mb]->[12.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[224.2mb]->[3.5mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s],
total [5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s],
total [6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[1.2mb]->[34.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total
[6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[266.2mb]->[21.9mb]/[266.2mb]}{[survivor]
[23.4mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total
[6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s],
total [5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young]
[246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s],
total [6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s],
total [6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s],
total [6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total
[7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
:

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624agdKkWdFCXtbEW%3DH4QJg8edsAuMZqF9ZpBZ%3DJeZew2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(erbdex) #5

How much RAM you need depends on how long you want to keep your data
around for. So, given you have ~200GB now on 4GB of RAM, you can probably
extrapolate that out based on your needs.

Isn't my problem more with 9G daily index, than with total of 200G(20
days x 9G) indexes?
Correct me if i am wrong here but doesn't kibana ask elasticsearch for just
one day/week of indices(based on the query).
Will elasticsearch really care if i have 500 days of total day-wise
segregated indices out there but am performing queries on just past 7 days?

Is this a total-footprint problem or a daily throughput problem?

On Monday, 12 May 2014 15:30:36 UTC+5:30, Mark Walkom wrote:

It's standard practise to use 50% of system memory for the heap.

How much RAM you need depends on how long you want to keep your data
around for. So, given you have ~200GB now on 4GB of RAM, you can probably
extrapolate that out based on your needs.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 12 May 2014 19:33, Abhishek Tiwari <erb...@gmail.com <javascript:>>wrote:

add more memory

i am doing 15 million docs, which total to ~9G. The average doc size is
~2KB.

  1. How much memory would you suggest for my use-case?
    2.Also, is it prudent for me to have half of OS memory dedicated to
    elasticsearch?

On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote:

You need to reduce your data size, add more memory or add another node.

Basically, you've reached the limits of that node.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 16:38, Abhishek Tiwari erb...@gmail.com wrote:

My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem).
Elasticsearch starts as-

498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10
/usr/bin/java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/
elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch
-cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/
share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.Elasticsearch

The node stopped responding (the ip:9200 status page), and so did
kibana. It started working fine on a restart.
i have logstash format docs wherein the index rotates daily.
Stats:
Daily: ~11G docs, ~15 million.
Total: 195G docs, ~300 million.

The logs of the time when it stopped responding are-

[2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[150.3mb]->[1.7mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s],
total [5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[197.4mb]->[9.3mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.5gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total
[6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[134.7mb]->[9.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[165.1mb]->[2.7mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s],
total [6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[265.9mb]->[2.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s],
total [5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[176.5mb]->[5.8mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s],
total [5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[167.4mb]->[12.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[224.2mb]->[3.5mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s],
total [5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s],
total [6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[1.2mb]->[34.7mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total
[6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[266.2mb]->[21.9mb]/[266.2mb]}{[survivor]
[23.4mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total
[6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s],
total [5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young]
[246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s],
total [6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s],
total [6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s],
total [6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total
[7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
:

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #6

Yes, kibana will load for whatever you ask for, but ES has to maintain
index metadata for every index in memory.
Those two coupled are pushing things too far for your heap.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 20:45, Abhishek Tiwari erbdex@gmail.com wrote:

How much RAM you need depends on how long you want to keep your data

around for. So, given you have ~200GB now on 4GB of RAM, you can probably
extrapolate that out based on your needs.

Isn't my problem more with 9G daily index, than with total of 200G(20
days x 9G) indexes?
Correct me if i am wrong here but doesn't kibana ask elasticsearch for
just one day/week of indices(based on the query).
Will elasticsearch really care if i have 500 days of total day-wise
segregated indices out there but am performing queries on just past 7
days
?
Is this a total-footprint problem or a daily throughput problem?

On Monday, 12 May 2014 15:30:36 UTC+5:30, Mark Walkom wrote:

It's standard practise to use 50% of system memory for the heap.

How much RAM you need depends on how long you want to keep your data
around for. So, given you have ~200GB now on 4GB of RAM, you can probably
extrapolate that out based on your needs.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 19:33, Abhishek Tiwari erb...@gmail.com wrote:

add more memory

i am doing 15 million docs, which total to ~9G. The average doc size is
~2KB.

  1. How much memory would you suggest for my use-case?
    2.Also, is it prudent for me to have half of OS memory dedicated to
    elasticsearch?

On Monday, 12 May 2014 14:03:19 UTC+5:30, Mark Walkom wrote:

You need to reduce your data size, add more memory or add another node.

Basically, you've reached the limits of that node.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 12 May 2014 16:38, Abhishek Tiwari erb...@gmail.com wrote:

My elasticsearch node is a AWS EC2 c3.xlarge (7.5G mem).
Elasticsearch starts as-

498 31810 99.6 64.6 163846656 4976944 ? Sl 06:03 26:10
/usr/bin/java -Xms4g -Xmx4g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/
elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch
-cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/s
hare/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/
elasticsearch org.elasticsearch.bootstrap.Elasticsearch

The node stopped responding (the ip:9200 status page), and so did
kibana. It started working fine on a restart.
i have logstash format docs wherein the index rotates daily.
Stats:
Daily: ~11G docs, ~15 million.
Total: 195G docs, ~300 million.

The logs of the time when it stopped responding are-

[2014-05-12 03:39:08,789][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:40:52,293][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240428][35773] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[150.3mb]->[1.7mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 03:44:11,739][INFO ][cluster.metadata ] [Hannibal
King] [logstash-2014.05.12] update_mapping [medusa_ex] (dynamic)
[2014-05-12 03:45:32,191][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][240703][35812] duration [5.2s], collections [1]/[5.8s],
total [5.2s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[197.4mb]->[9.3mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.5gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:06:01,224][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][241926][35985] duration [6s], collections [1]/[6.2s], total
[6s]/[4.7h], memory [3.7gb]->[3.6gb]/[3.9gb], all_pools {[young]
[134.7mb]->[9.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:08:14,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242049][36004] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[165.1mb]->[2.7mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:09:07,473][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242096][36011] duration [6.2s], collections [1]/[6.7s],
total [6.2s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[265.9mb]->[2.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:10:08,387][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242152][36020] duration [5.4s], collections [1]/[5.6s],
total [5.4s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[176.5mb]->[5.8mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:13:12,774][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242326][36046] duration [5.6s], collections [1]/[5.8s],
total [5.6s]/[4.7h], memory [3.8gb]->[3.5gb]/[3.9gb], all_pools {[young]
[167.4mb]->[12.9mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.5gb]/[3.6gb]}
[2014-05-12 04:14:22,729][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242386][36057] duration [6.3s], collections [1]/[6.5s],
total [6.3s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[224.2mb]->[3.5mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:12,192][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242431][36064] duration [5.2s], collections [1]/[5.4s],
total [5.2s]/[4.7h], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young]
[234mb]->[2.4mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:32,344][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242445][36067] duration [6.3s], collections [1]/[7.1s],
total [6.3s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[1.2mb]->[34.7mb]/[266.2mb]}{[survivor]
[33.2mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:39,627][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242446][36068] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.7h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[34.7mb]->[45.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:15:51,547][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242448][36070] duration [6.5s], collections [1]/[7s], total
[6.5s]/[4.7h], memory [3.6gb]->[3.7gb]/[3.9gb], all_pools {[young]
[24.4mb]->[32.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:03,300][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242451][36072] duration [5.8s], collections [1]/[5.9s],
total [5.8s]/[4.7h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young]
[266.2mb]->[21.9mb]/[266.2mb]}{[survivor]
[23.4mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:12,454][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242454][36073] duration [6.6s], collections [1]/[7s], total
[6.6s]/[4.7h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[196.8mb]->[33.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:24,635][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242457][36075] duration [5.7s], collections [1]/[5.8s],
total [5.7s]/[4.8h], memory [3.9gb]->[3.7gb]/[3.9gb], all_pools {[young]
[246.4mb]->[44.3mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:32,239][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242459][36076] duration [6.4s], collections [1]/[6.6s],
total [6.4s]/[4.8h], memory [3.8gb]->[3.7gb]/[3.9gb], all_pools {[young]
[221.5mb]->[31.9mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:43,990][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242461][36078] duration [6.4s], collections [1]/[6.9s],
total [6.4s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[40.5mb]->[64.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:51,259][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242462][36079] duration [6.6s], collections [1]/[7.2s],
total [6.6s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[64.2mb]->[74.6mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:16:58,791][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242463][36080] duration [7s], collections [1]/[7.5s], total
[7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[74.6mb]->[89.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
[2014-05-12 04:17:06,028][INFO ][monitor.jvm ] [Hannibal
King] [gc][old][242464][36081] duration [6.7s], collections [1]/[7.2s],
total [6.7s]/[4.8h], memory [3.7gb]->[3.7gb]/[3.9gb], all_pools {[young]
[89.7mb]->[93.2mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old]
[3.6gb]->[3.6gb]/[3.6gb]}
:

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2019fb64-e252-446d-aee9-b8881ed7cd92%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5387deb7-e3b8-4888-bf83-ecec918c0d70%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/ec6f20ad-af62-4a01-9f2b-6df54d2d2568%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bT-DR0oYaAcHOxgR0MM7SdoEM%3DXw_GcRD9aGZT4uT4gA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #7