Questions about memory usage

Elastic Stack Elasticsearch
1

Hi,

I am using ElasticSearch for a few months in a project. The simplicity and
power it has is.. really amazing.
On this project we are using a VPS to host a normal LAMP stack +
ElasticSearch.

The problem is the memory usage.
Without anything the server is using: (cumulative values)
Nothing: 28MB
With MySQL: 159 MB
With Apache: 216 MB
With ES: 650~700MB

The thing is I'm only indexing around 10k documents. My config file (running
as a service) is:

set.default.ES_HOME=
set.default.ES_MIN_MEM=32
set.default.ES_MAX_MEM=64

htop screenshot: http://imgur.com/kC3W9

I've checked and the server is started with the correct memory
configuration, but still after a couple of hours, 2-3 days the memory usage
just goes over the top and the server crashes.
So I need to be constantly restart the elastic search service.

I'm running a CentOS on a VPS with a limit of 1GB. Any more thing you would
like to know ask please.
Even tough this is nothing VERY serious right now, I'm worried that with
traffic spike the server would crash/restart very quickly. Can you help me?

Thanks,
Diogo

2

Which version of elasticsearch are you using?

On Fri, Jul 29, 2011 at 5:12 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Hi,

I am using Elasticsearch for a few months in a project. The simplicity and
power it has is.. really amazing.
On this project we are using a VPS to host a normal LAMP stack +
Elasticsearch.

The problem is the memory usage.
Without anything the server is using: (cumulative values)
Nothing: 28MB
With MySQL: 159 MB
With Apache: 216 MB
With ES: 650~700MB

The thing is I'm only indexing around 10k documents. My config file
(running as a service) is:

set.default.ES_HOME=
set.default.ES_MIN_MEM=32
set.default.ES_MAX_MEM=64

htop screenshot: Imgur: The magic of the Internet

I've checked and the server is started with the correct memory
configuration, but still after a couple of hours, 2-3 days the memory usage
just goes over the top and the server crashes.
So I need to be constantly restart the Elasticsearch service.

I'm running a CentOS on a VPS with a limit of 1GB. Any more thing you would
like to know ask please.
Even tough this is nothing VERY serious right now, I'm worried that with
traffic spike the server would crash/restart very quickly. Can you help me?

Thanks,
Diogo

3

Good question, how can I check the version?

4

It prints the version in hte log when it starts up.

On Fri, Jul 29, 2011 at 5:42 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Good question, how can I check the version?

5

0.16.0. Should I update it and then see what happens?
To update its just replace the existing files and restart the service?

6

Updated to the latest version, updated the index, but still the huge memory
footprint.

7

Do you have any idea?

8

Whats the RES memory that you see for the process?

On Tue, Aug 2, 2011 at 9:50 PM, Diogo Figueiredo diogoepf@gmail.com wrote:

Do you have any idea?

9

Normaly around 149MB. If I kill all processes and restart ES it will go to
100MB (only ES and basic linux processes). But Virtual is around 400MB.

What do you think?

On Tue, Aug 2, 2011 at 8:53 PM, Shay Banon kimchy@gmail.com wrote:

Whats the RES memory that you see for the process?

On Tue, Aug 2, 2011 at 9:50 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Do you have any idea?

10

Sounds sensible to me

On Tue, Aug 2, 2011 at 10:03 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Normaly around 149MB. If I kill all processes and restart ES it will go to
100MB (only ES and basic linux processes). But Virtual is around 400MB.

What do you think?

On Tue, Aug 2, 2011 at 8:53 PM, Shay Banon kimchy@gmail.com wrote:

Whats the RES memory that you see for the process?

On Tue, Aug 2, 2011 at 9:50 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Do you have any idea?

11

I will look somewhere else then :wink: Thanks for the fast replies :slight_smile:

On Tue, Aug 2, 2011 at 9:05 PM, Shay Banon kimchy@gmail.com wrote:

Sounds sensible to me

On Tue, Aug 2, 2011 at 10:03 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Normaly around 149MB. If I kill all processes and restart ES it will go to
100MB (only ES and basic linux processes). But Virtual is around 400MB.

What do you think?

On Tue, Aug 2, 2011 at 8:53 PM, Shay Banon kimchy@gmail.com wrote:

Whats the RES memory that you see for the process?

On Tue, Aug 2, 2011 at 9:50 PM, Diogo Figueiredo diogoepf@gmail.comwrote:

Do you have any idea?

12

Getting back to this.. Even if my Max memory is 64MB today it reached to
more than 300MB Res memory. Is that normal?
Because I set it up to Max 64 MB so I would expect it not to cross that
point..

13

When you set the max heap memory to 64mb, thats what it will control, its
the heap memory the java process will have. It does not count memory
associated with things like sockets, file handles and so on.

On Fri, Aug 5, 2011 at 2:15 AM, Diogo Figueiredo diogoepf@gmail.com wrote:

Getting back to this.. Even if my Max memory is 64MB today it reached to
more than 300MB Res memory. Is that normal?
Because I set it up to Max 64 MB so I would expect it not to cross that
point..