We have a 2 node elasticsearch cluster which is used by logstash to store
log files. The current input is around 100 documents (logs) per second wit
a size of around 50kb - 150kb.
Compared to what i have read so far this is not a high amount but we
experience already a high heap usage 70% form the total of 11GB heap size,
the system has in total 32GB RAM. CPU and IO are totally fine.
We have a 2 node elasticsearch cluster which is used by logstash to store
log files. The current input is around 100 documents (logs) per second wit
a size of around 50kb - 150kb.
Compared to what i have read so far this is not a high amount but we
experience already a high heap usage 70% form the total of 11GB heap size,
the system has in total 32GB RAM. CPU and IO are totally fine.
This is not high. The JVM always uses the whole heap to avoid garbage
collection as much as possible. In ES, a threshold is set to 75% before CMS
garbage collection kicks in.
We have a 2 node elasticsearch cluster which is used by logstash to store
log files. The current input is around 100 documents (logs) per second wit
a size of around 50kb - 150kb.
Compared to what i have read so far this is not a high amount but we
experience already a high heap usage 70% form the total of 11GB heap size,
the system has in total 32GB RAM. CPU and IO are totally fine.
Thanks for your answer! We are using the default values, so no doc_values.
I did some research about it and it sounds very interesting and helpful to
keep the heap usage lower.
How can I add doc_values: true to the index template so that the new daily
based generated indexes using this feature.
Cheers
Chris
On Monday, March 16, 2015 at 11:36:13 AM UTC+7, Mark Walkom wrote:
Those are reasonably large documents. You also seem to have a lot of
shards for the data.
What sort of data is it, are you using doc values, how are you bucketing
data (ie time series indices)?
We have a 2 node elasticsearch cluster which is used by logstash to store
log files. The current input is around 100 documents (logs) per second wit
a size of around 50kb - 150kb.
Compared to what i have read so far this is not a high amount but we
experience already a high heap usage 70% form the total of 11GB heap size,
the system has in total 32GB RAM. CPU and IO are totally fine.
Thanks for your answer! We are using the default values, so no doc_values.
I did some research about it and it sounds very interesting and helpful to
keep the heap usage lower.
How can I add doc_values: true to the index template so that the new daily
based generated indexes using this feature.
Cheers
Chris
On Monday, March 16, 2015 at 11:36:13 AM UTC+7, Mark Walkom wrote:
Those are reasonably large documents. You also seem to have a lot of
shards for the data.
What sort of data is it, are you using doc values, how are you bucketing
data (ie time series indices)?
We have a 2 node elasticsearch cluster which is used by logstash to
store log files. The current input is around 100 documents (logs) per
second wit a size of around 50kb - 150kb.
Compared to what i have read so far this is not a high amount but we
experience already a high heap usage 70% form the total of 11GB heap size,
the system has in total 32GB RAM. CPU and IO are totally fine.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
