Very small ES instance consuming too much memory


(Rubish Gupta) #1

Hi,

I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.

The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?


(Karussell) #2

did you specify the maximum and minimum memory?

http://www.elasticsearch.org/tutorials/2010/07/02/setting-up-elasticsearch-on-debian.html

it should be

ES_MIN_MEM=256m
ES_MAX_MEM=256m

as approx the half of the available memory should be free for the OS
itself (to allow caching on OS-side)

And then only if ES requires more memory it'll throw OutOfMemoryError.
then you should either decrease memory usage or increase RAM :wink:

On 22 Jul., 10:31, rubish gupta rubish.gu...@almaconnect.com wrote:

Hi,

I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.

The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?


(Rubish Gupta) #3

Yes I did use min/max memory, but I used MIN=256 and MAX=400. I will
decrease the max memory.

But even then the overall stats on machine look a lot higher than what I
have configured.

Are there any other commands on OS(ubuntu) or ES I can run to see exactly
how much memory is taken by ES?

On Fri, Jul 22, 2011 at 8:01 PM, Karussell tableyourtime@googlemail.comwrote:

did you specify the maximum and minimum memory?

http://www.elasticsearch.org/tutorials/2010/07/02/setting-up-elasticsearch-on-debian.html

it should be

ES_MIN_MEM=256m
ES_MAX_MEM=256m

as approx the half of the available memory should be free for the OS
itself (to allow caching on OS-side)

And then only if ES requires more memory it'll throw OutOfMemoryError.
then you should either decrease memory usage or increase RAM :wink:

On 22 Jul., 10:31, rubish gupta rubish.gu...@almaconnect.com wrote:

Hi,

I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably
it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.

The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First thing
I
will do is decrease the number of shards from 5 to 1. Next I am planning
to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?


(danoyoung) #4

Check out jvisualvm or jmap

Regards,

Dan
On Jul 22, 2011 4:06 PM, "rubish gupta" rubish.gupta@almaconnect.com
wrote:

Yes I did use min/max memory, but I used MIN=256 and MAX=400. I will
decrease the max memory.

But even then the overall stats on machine look a lot higher than what I
have configured.

Are there any other commands on OS(ubuntu) or ES I can run to see exactly
how much memory is taken by ES?

On Fri, Jul 22, 2011 at 8:01 PM, Karussell <tableyourtime@googlemail.com
wrote:

did you specify the maximum and minimum memory?

http://www.elasticsearch.org/tutorials/2010/07/02/setting-up-elasticsearch-on-debian.html

it should be

ES_MIN_MEM=256m
ES_MAX_MEM=256m

as approx the half of the available memory should be free for the OS
itself (to allow caching on OS-side)

And then only if ES requires more memory it'll throw OutOfMemoryError.
then you should either decrease memory usage or increase RAM :wink:

On 22 Jul., 10:31, rubish gupta rubish.gu...@almaconnect.com wrote:

Hi,

I have a small application which is growing slowly. I only index user
and

posts data in ES for now. I haven't configured shrads/index, so
probably

it

is using default of 5 shards/index. I have ~6k users and <1k posts till
now.

The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First
thing

I

will do is decrease the number of shards from 5 to 1. Next I am
planning

to

migrate to ES 0.17.1(currently I am using 0.15.2), but it might take
some

time before I migrate. What do you guys suggest I can do in meanwhile
to

control this memory overflow?


(Rubish Gupta) #5

I have gisted the output of jmap, top and free at
https://gist.github.com/1100776. Can somebody help me get some meaning
attached to the outputs. Right now it is just some lines of text which I
can't comprehend.

On Sat, Jul 23, 2011 at 5:14 AM, Dan Young danoyoung@gmail.com wrote:

Check out jvisualvm or jmap

Regards,

Dan
On Jul 22, 2011 4:06 PM, "rubish gupta" rubish.gupta@almaconnect.com
wrote:

Yes I did use min/max memory, but I used MIN=256 and MAX=400. I will
decrease the max memory.

But even then the overall stats on machine look a lot higher than what I
have configured.

Are there any other commands on OS(ubuntu) or ES I can run to see exactly
how much memory is taken by ES?

On Fri, Jul 22, 2011 at 8:01 PM, Karussell <tableyourtime@googlemail.com
wrote:

did you specify the maximum and minimum memory?

http://www.elasticsearch.org/tutorials/2010/07/02/setting-up-elasticsearch-on-debian.html

it should be

ES_MIN_MEM=256m
ES_MAX_MEM=256m

as approx the half of the available memory should be free for the OS
itself (to allow caching on OS-side)

And then only if ES requires more memory it'll throw OutOfMemoryError.
then you should either decrease memory usage or increase RAM :wink:

On 22 Jul., 10:31, rubish gupta rubish.gu...@almaconnect.com wrote:

Hi,

I have a small application which is growing slowly. I only index user
and

posts data in ES for now. I haven't configured shrads/index, so
probably

it

is using default of 5 shards/index. I have ~6k users and <1k posts
till

now.

The server is a 512MB server, only running ES. Right now htop on
server

machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First
thing

I

will do is decrease the number of shards from 5 to 1. Next I am
planning

to

migrate to ES 0.17.1(currently I am using 0.15.2), but it might take
some

time before I migrate. What do you guys suggest I can do in meanwhile
to

control this memory overflow?


(Shay Banon) #6

Even if you specify MIN and MAX memory, those apply to the JVM heap size,
there is still an overhead associated with the java process itself, though,
based on your usage, it should be constant so you can extrapolate what it
is.

On Sat, Jul 23, 2011 at 3:48 AM, rubish gupta
rubish.gupta@almaconnect.comwrote:

I have gisted the output of jmap, top and free at
https://gist.github.com/1100776. Can somebody help me get some meaning
attached to the outputs. Right now it is just some lines of text which I
can't comprehend.

On Sat, Jul 23, 2011 at 5:14 AM, Dan Young danoyoung@gmail.com wrote:

Check out jvisualvm or jmap

Regards,

Dan
On Jul 22, 2011 4:06 PM, "rubish gupta" rubish.gupta@almaconnect.com
wrote:

Yes I did use min/max memory, but I used MIN=256 and MAX=400. I will
decrease the max memory.

But even then the overall stats on machine look a lot higher than what I
have configured.

Are there any other commands on OS(ubuntu) or ES I can run to see
exactly
how much memory is taken by ES?

On Fri, Jul 22, 2011 at 8:01 PM, Karussell <
tableyourtime@googlemail.com>wrote:

did you specify the maximum and minimum memory?

http://www.elasticsearch.org/tutorials/2010/07/02/setting-up-elasticsearch-on-debian.html

it should be

ES_MIN_MEM=256m
ES_MAX_MEM=256m

as approx the half of the available memory should be free for the OS
itself (to allow caching on OS-side)

And then only if ES requires more memory it'll throw OutOfMemoryError.
then you should either decrease memory usage or increase RAM :wink:

On 22 Jul., 10:31, rubish gupta rubish.gu...@almaconnect.com wrote:

Hi,

I have a small application which is growing slowly. I only index user
and

posts data in ES for now. I haven't configured shrads/index, so
probably

it

is using default of 5 shards/index. I have ~6k users and <1k posts
till

now.

The server is a 512MB server, only running ES. Right now htop on
server

machine shows ES is consuming 1531MB of virtual memory. I do not have
much
experience in JAVA, so am not familiar with tools to monitor JVM and
such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First
thing

I

will do is decrease the number of shards from 5 to 1. Next I am
planning

to

migrate to ES 0.17.1(currently I am using 0.15.2), but it might take
some

time before I migrate. What do you guys suggest I can do in meanwhile
to

control this memory overflow?


(ofavre) #7

Are you talking of virtual memory?
This has nothing to do with the real memory usage, ie resident memory (if
you have no swapping).

Let me explain:
Let's assume you have a 100 GB index stored on the disk, and for performance
reasons, you are using index.store.type=mmap.
Let's assume then that you have successfully configured ES to use 10 GB of
RAM.
You would end up having a memory usage as follows:
RES (resident memory) = 10.4 GB
VIRT (virtual memory) = 110.4 GB
And I am very far from having that much memory in RAM and in the swap !
The virtual memory used by mmap is... virtual. It does a projection of the
files into the process memory addressing, but the OS under the hood, only
mounts parts of the file into memory, as it sees can fit into the cache.
(well, I'm not 100% sure of that...)
This is also why you should let some RAM to the OS for its caching.

Remember: virtual memory is... strange, and does not reflect the RAM usage.
Look at the resident memory.
(Not sure where the swap usage stands, and I now that there is no exact
calculation of that under linux given the resident and virtual memory
usage).

Cheers,

Olivier Favre

2011/7/22 rubish gupta rubish.gupta@almaconnect.com

Hi,

I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.

The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?


(Rubish Gupta) #8

Thanks for the clarification on virtual memory. It makes much more sense
now.

On Mon, Jul 25, 2011 at 1:27 PM, Olivier Favre olivier@yakaz.com wrote:

Are you talking of virtual memory?
This has nothing to do with the real memory usage, ie resident memory
(if you have no swapping).

Let me explain:
Let's assume you have a 100 GB index stored on the disk, and for
performance reasons, you are using index.store.type=mmap.
Let's assume then that you have successfully configured ES to use 10 GB of
RAM.
You would end up having a memory usage as follows:
RES (resident memory) = 10.4 GB
VIRT (virtual memory) = 110.4 GB
And I am very far from having that much memory in RAM and in the swap !
The virtual memory used by mmap is... virtual. It does a projection of the
files into the process memory addressing, but the OS under the hood, only
mounts parts of the file into memory, as it sees can fit into the cache.
(well, I'm not 100% sure of that...)
This is also why you should let some RAM to the OS for its caching.

Remember: virtual memory is... strange, and does not reflect the RAM usage.
Look at the resident memory.
(Not sure where the swap usage stands, and I now that there is no exact
calculation of that under linux given the resident and virtual memory
usage).

Cheers,

Olivier Favre

www.yakaz.com

2011/7/22 rubish gupta rubish.gupta@almaconnect.com

Hi,

I have a small application which is growing slowly. I only index user and
posts data in ES for now. I haven't configured shrads/index, so probably it
is using default of 5 shards/index. I have ~6k users and <1k posts till
now.

The server is a 512MB server, only running ES. Right now htop on server
machine shows ES is consuming 1531MB of virtual memory. I do not have much
experience in JAVA, so am not familiar with tools to monitor JVM and such,
so pardon me if info I provide is insufficient. Here is the output of
running free -m on the machine.

deploy@elastic1:~$ free -m
total used free shared buffers cached
Mem: 496 488 8 0 5 23
-/+ buffers/cache: 459 36
Swap: 1023 568 455

I guess it should not be using such a huge amount of memory. First thing I
will do is decrease the number of shards from 5 to 1. Next I am planning to
migrate to ES 0.17.1(currently I am using 0.15.2), but it might take some
time before I migrate. What do you guys suggest I can do in meanwhile to
control this memory overflow?


(system) #9