I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.
The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?
I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.
The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?
I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably
it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.
The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First thing
I
will do is decrease the number of shards from 5 to 1. Next I am planning
to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?
I have a small application which is growing slowly. I only index user
and
posts data in ES for now. I haven't configured shrads/index, so
probably
it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.
The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First
thing
I
will do is decrease the number of shards from 5 to 1. Next I am
planning
to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take
some
time before I migrate. What do you guys suggest I can do in meanwhile
to
control this memory overflow?
I have gisted the output of jmap, top and free at ES memory usage · GitHub. Can somebody help me get some meaning
attached to the outputs. Right now it is just some lines of text which I
can't comprehend.
I have a small application which is growing slowly. I only index user
and
posts data in ES for now. I haven't configured shrads/index, so
probably
it
is using default of 5 shards/index. I have ~6k users and <1k posts
till
now.
The server is a 512MB server, only running ES. Right now htop on
server
machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First
thing
I
will do is decrease the number of shards from 5 to 1. Next I am
planning
to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take
some
time before I migrate. What do you guys suggest I can do in meanwhile
to
control this memory overflow?
Even if you specify MIN and MAX memory, those apply to the JVM heap size,
there is still an overhead associated with the java process itself, though,
based on your usage, it should be constant so you can extrapolate what it
is.
I have gisted the output of jmap, top and free at ES memory usage · GitHub. Can somebody help me get some meaning
attached to the outputs. Right now it is just some lines of text which I
can't comprehend.
I have a small application which is growing slowly. I only index user
and
posts data in ES for now. I haven't configured shrads/index, so
probably
it
is using default of 5 shards/index. I have ~6k users and <1k posts
till
now.
The server is a 512MB server, only running ES. Right now htop on
server
machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First
thing
I
will do is decrease the number of shards from 5 to 1. Next I am
planning
to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take
some
time before I migrate. What do you guys suggest I can do in meanwhile
to
control this memory overflow?
Are you talking of virtual memory?
This has nothing to do with the real memory usage, ie resident memory (if
you have no swapping).
Let me explain:
Let's assume you have a 100 GB index stored on the disk, and for performance
reasons, you are using index.store.type=mmap.
Let's assume then that you have successfully configured ES to use 10 GB of
RAM.
You would end up having a memory usage as follows:
RES (resident memory) = 10.4 GB
VIRT (virtual memory) = 110.4 GB
And I am very far from having that much memory in RAM and in the swap !
The virtual memory used by mmap is... virtual. It does a projection of the
files into the process memory addressing, but the OS under the hood, only
mounts parts of the file into memory, as it sees can fit into the cache.
(well, I'm not 100% sure of that...)
This is also why you should let some RAM to the OS for its caching.
Remember: virtual memory is... strange, and does not reflect the RAM usage.
Look at the resident memory.
(Not sure where the swap usage stands, and I now that there is no exact
calculation of that under linux given the resident and virtual memory
usage).
I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.
The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?
Thanks for the clarification on virtual memory. It makes much more sense
now.
On Mon, Jul 25, 2011 at 1:27 PM, Olivier Favre olivier@yakaz.com wrote:
Are you talking of virtual memory?
This has nothing to do with the real memory usage, ie resident memory
(if you have no swapping).
Let me explain:
Let's assume you have a 100 GB index stored on the disk, and for
performance reasons, you are using index.store.type=mmap.
Let's assume then that you have successfully configured ES to use 10 GB of
RAM.
You would end up having a memory usage as follows:
RES (resident memory) = 10.4 GB
VIRT (virtual memory) = 110.4 GB
And I am very far from having that much memory in RAM and in the swap !
The virtual memory used by mmap is... virtual. It does a projection of the
files into the process memory addressing, but the OS under the hood, only
mounts parts of the file into memory, as it sees can fit into the cache.
(well, I'm not 100% sure of that...)
This is also why you should let some RAM to the OS for its caching.
Remember: virtual memory is... strange, and does not reflect the RAM usage.
Look at the resident memory.
(Not sure where the swap usage stands, and I now that there is no exact
calculation of that under linux given the resident and virtual memory
usage).
I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.
The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.
I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
