Memory problems

Elastic Stack Elasticsearch
1

Any help appreciated diagnosing this problem, it's been bothering me
for days.

Here's a graph take from the last months memory usage: http://imgur.com/7FnJN
with annoations

  1. The server was restarted. ElasticSearch is running OpenJDK at this
    point.
  2. The app memory balloons and Elastic is restarted.
  3. Same as before.
  4. Switched to Suns JRE and now the app memory is under control, but
    the swap follows a very similar pattern to that of the memory when
    using OpenJDK

ElasticSearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf - http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml - http://pastie.org/private/aarhchbycpax8lwczjswa

Version: 0.16.2

Dataset

Around 3,000 entries with 4 string fields, 2 ints, 1 float and a date
time field. I would expect such a small data set to not use the amount
of memory elastic is using.

System

Total memory = 469M
vm.swappiness = 10

Java

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20
  • A frequent search term error I'm not too worried about, but could be
    a contributor: http://pastie.org/private/ceuoczbn65natwg6epm2a

If any more information is required then I'd be happy to help out.

Thanks for reading.

2

It appears the two links for the configuration files are broken:

bin/service/elasticsearch.conf -
http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -
http://pastie.org/private/aarhchbycpax8lwczjswa

Should work okay.

Kind Regards,
-Mathew Davies.

On 20 June 2011 19:25, Mathew Davies thepixeldeveloper@googlemail.comwrote:

Any help appreciated diagnosing this problem, it's been bothering me
for days.

Here's a graph take from the last months memory usage:
Imgur: The magic of the Internet
with annoations

  1. The server was restarted. Elasticsearch is running OpenJDK at this
    point.
  2. The app memory balloons and Elastic is restarted.
  3. Same as before.
  4. Switched to Suns JRE and now the app memory is under control, but
    the swap follows a very similar pattern to that of the memory when
    using OpenJDK

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf -
http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -
http://pastie.org/private/aarhchbycpax8lwczjswa

Version: 0.16.2

Dataset

Around 3,000 entries with 4 string fields, 2 ints, 1 float and a date
time field. I would expect such a small data set to not use the amount
of memory elastic is using.

System

Total memory = 469M
vm.swappiness = 10

Java

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20
  • A frequent search term error I'm not too worried about, but could be
    a contributor: http://pastie.org/private/ceuoczbn65natwg6epm2a

If any more information is required then I'd be happy to help out.

Thanks for reading.

3

Hi Matt

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf - http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml - http://pastie.org/private/aarhchbycpax8lwczjswa

Are you sure that your mlockall is having any effect?

Are you sure that the user under which ES is running has ulimit -l
unlimited? If it isn't unlimited, then your mlockall will be silently
ignored.

Your ES_MIN/MAX_MEM are set to 64MB, which is pretty low - makes it
difficult to distinguish how much of your memory is assigned to the heap
and how much to non-heap.

I'd download GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster. - makes it easier to
see where your memory is being used.

Is ES the only thing on the box? Are you sure it is ES that is using the
swap?

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20

Don't recognise this one

That is just a malformed query string query, because you have a colon
without a field name. You may want to try using the 'text' query
instead.

clint

4

Hi Clinton.

I have enabled ulimit -l unlimited, but failed to mention it in the original
post. Apologies

You can see the effect it's having here: Imgur: The magic of the Internet

I'm keeping a very close eye on the swap and it's increasing as it was
before.

I'd download GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster. - makes it easier to

see where your memory is being used.

Not heard of this, I'll download it and take a look.

That is just a malformed query string query, because you have a colon

without a field name. You may want to try using the 'text' query
instead.

-Mathew Davies.

On 20 June 2011 19:44, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Matt

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf -
http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -
http://pastie.org/private/aarhchbycpax8lwczjswa

Are you sure that your mlockall is having any effect?

Are you sure that the user under which ES is running has ulimit -l
unlimited? If it isn't unlimited, then your mlockall will be silently
ignored.

Your ES_MIN/MAX_MEM are set to 64MB, which is pretty low - makes it
difficult to distinguish how much of your memory is assigned to the heap
and how much to non-heap.

I'd download GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster. - makes it easier to
see where your memory is being used.

Is ES the only thing on the box? Are you sure it is ES that is using the
swap?

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20

Don't recognise this one

That is just a malformed query string query, because you have a colon
without a field name. You may want to try using the 'text' query
instead.

clint

5

We started using ES with OpenJDK and were convinced to use the Sun JDK
due to some nuanced differences that have potential to cause problems.
Not sure if that is the case here, but I'd not recommend using
OpenJDK.

Best Regards,
Paul

On Jun 20, 12:44 pm, Clinton Gormley clin...@iannounce.co.uk wrote:

Hi Matt

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf -http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -http://pastie.org/private/aarhchbycpax8lwczjswa

Are you sure that your mlockall is having any effect?

Are you sure that the user under which ES is running has ulimit -l
unlimited? If it isn't unlimited, then your mlockall will be silently
ignored.

Your ES_MIN/MAX_MEM are set to 64MB, which is pretty low - makes it
difficult to distinguish how much of your memory is assigned to the heap
and how much to non-heap.

I'd downloadhttps://github.com/lukas-vlcek/bigdesk- makes it easier to
see where your memory is being used.

Is ES the only thing on the box? Are you sure it is ES that is using the
swap?

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20

Don't recognise this one

That is just a malformed query string query, because you have a colon
without a field name. You may want to try using the 'text' query
instead.

clint

6

I mentioned in point 4 that I had ditched OpenJDK and switched to Suns
implementation. I assume you were mentioning it for everyone elses sake
rather than me.

Thanks.

-Mathew Davies.

On 20 June 2011 19:57, Paul ppearcy@gmail.com wrote:

We started using ES with OpenJDK and were convinced to use the Sun JDK
due to some nuanced differences that have potential to cause problems.
Not sure if that is the case here, but I'd not recommend using
OpenJDK.

Best Regards,
Paul

On Jun 20, 12:44 pm, Clinton Gormley clin...@iannounce.co.uk wrote:

Hi Matt

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf -
http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -
http://pastie.org/private/aarhchbycpax8lwczjswa

Are you sure that your mlockall is having any effect?

Are you sure that the user under which ES is running has ulimit -l
unlimited? If it isn't unlimited, then your mlockall will be silently
ignored.

Your ES_MIN/MAX_MEM are set to 64MB, which is pretty low - makes it
difficult to distinguish how much of your memory is assigned to the heap
and how much to non-heap.

I'd downloadhttps://github.com/lukas-vlcek/bigdesk- makes it easier to
see where your memory is being used.

Is ES the only thing on the box? Are you sure it is ES that is using the
swap?

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20

Don't recognise this one

That is just a malformed query string query, because you have a colon
without a field name. You may want to try using the 'text' query
instead.

clint

7

I forget to answer your questions Clinton.

Is ES the only thing on the box?

No; I have MySQL, nginx and PHP-FPM running alongside, although they use
very little memory.

Are you sure it is ES that is using the swap?

Almost certain. I never had issues with swap before running Elasticsearch. I
can't say for definite though as I only started using munin after I switched
to Elastic.

Here's an updated munin screenshot: Imgur: The magic of the Internet You can see where
it's allocated the memory properly this time thanks to "ulimit -l
unlimited".

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

I'm too familiar with the JVMs memory management, so if someone could help
me understand the values it'd be much appreciated.

-Mathew Davies.

On 20 June 2011 20:06, Mathew Davies thepixeldeveloper@googlemail.comwrote:

I mentioned in point 4 that I had ditched OpenJDK and switched to Suns
implementation. I assume you were mentioning it for everyone elses sake
rather than me.

Thanks.

-Mathew Davies.

On 20 June 2011 19:57, Paul ppearcy@gmail.com wrote:

We started using ES with OpenJDK and were convinced to use the Sun JDK
due to some nuanced differences that have potential to cause problems.
Not sure if that is the case here, but I'd not recommend using
OpenJDK.

Best Regards,
Paul

On Jun 20, 12:44 pm, Clinton Gormley clin...@iannounce.co.uk wrote:

Hi Matt

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf -
http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -
http://pastie.org/private/aarhchbycpax8lwczjswa

Are you sure that your mlockall is having any effect?

Are you sure that the user under which ES is running has ulimit -l
unlimited? If it isn't unlimited, then your mlockall will be silently
ignored.

Your ES_MIN/MAX_MEM are set to 64MB, which is pretty low - makes it
difficult to distinguish how much of your memory is assigned to the heap
and how much to non-heap.

I'd downloadhttps://github.com/lukas-vlcek/bigdesk- makes it easier to
see where your memory is being used.

Is ES the only thing on the box? Are you sure it is ES that is using the
swap?

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20

Don't recognise this one

That is just a malformed query string query, because you have a colon
without a field name. You may want to try using the 'text' query
instead.

clint

8

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

9

Elasticsearch memory usage still grows when very little is happening. I've
taken two new screenshots:

Old BigDesk : http://i.imgur.com/Xz21Y.jpg
New BigDesk: http://i.imgur.com/vseGU.jpg

The non-heap memory has expanded. Mem heap is around the same.

Old Munin: Imgur: The magic of the Internet
New Munin: Imgur: The magic of the Internet

Elasticsearch memory has slowly risen over the last few days to point
where's it's now eating into the swap again. All configuration is the same
as when I posted it further up.

Any help is greatly appreciated.

-Mathew Davies.

On 21 June 2011 15:14, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

10

Are you using scripts when you search?

On Thursday, June 23, 2011 at 9:20 PM, Mathew Davies wrote:

Elasticsearch memory usage still grows when very little is happening. I've taken two new screenshots:

Old BigDesk : http://i.imgur.com/Xz21Y.jpg
New BigDesk: http://i.imgur.com/vseGU.jpg

The non-heap memory has expanded. Mem heap is around the same.

Old Munin: Imgur: The magic of the Internet
New Munin: Imgur: The magic of the Internet

Elasticsearch memory has slowly risen over the last few days to point where's it's now eating into the swap again. All configuration is the same as when I posted it further up.

Any help is greatly appreciated.

-Mathew Davies.

On 21 June 2011 15:14, Clinton Gormley <clinton@iannounce.co.uk (mailto:clinton@iannounce.co.uk)> wrote:

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

11

No.

On 24 June 2011 10:43, Shay Banon shay.banon@elasticsearch.com wrote:

Are you using scripts when you search?

On Thursday, June 23, 2011 at 9:20 PM, Mathew Davies wrote:

Elasticsearch memory usage still grows when very little is happening. I've
taken two new screenshots:

Old BigDesk : http://i.imgur.com/Xz21Y.jpg
New BigDesk: http://i.imgur.com/vseGU.jpg

The non-heap memory has expanded. Mem heap is around the same.

Old Munin: Imgur: The magic of the Internet
New Munin: Imgur: The magic of the Internet

Elasticsearch memory has slowly risen over the last few days to point
where's it's now eating into the swap again. All configuration is the same
as when I posted it further up.

Any help is greatly appreciated.

-Mathew Davies.

On 21 June 2011 15:14, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

12

The rise in non heap memory usage is quite small (at least based on what I can tell from the graph), and the heap usage is constant as well. Maybe you answered it and I missed it, but are you sure its the ES process?

On Friday, June 24, 2011 at 1:09 PM, Mathew Davies wrote:

No.

On 24 June 2011 10:43, Shay Banon <shay.banon@elasticsearch.com (mailto:shay.banon@elasticsearch.com)> wrote:

Are you using scripts when you search?

On Thursday, June 23, 2011 at 9:20 PM, Mathew Davies wrote:

Elasticsearch memory usage still grows when very little is happening. I've taken two new screenshots:

Old BigDesk : http://i.imgur.com/Xz21Y.jpg
New BigDesk: http://i.imgur.com/vseGU.jpg

The non-heap memory has expanded. Mem heap is around the same.

Old Munin: Imgur: The magic of the Internet
New Munin: Imgur: The magic of the Internet

Elasticsearch memory has slowly risen over the last few days to point where's it's now eating into the swap again. All configuration is the same as when I posted it further up.

Any help is greatly appreciated.

-Mathew Davies.

On 21 June 2011 15:14, Clinton Gormley <clinton@iannounce.co.uk (mailto:clinton@iannounce.co.uk)> wrote:

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

13

The heap is now growing at a steady pace. In the first big-desk screenshot
you can see it topping it off at around 30MB (only 16 hours ago). It's
bordering on 50MB now.

Imgur

I can't take a look at the usage over time because of the way big-desk works
and can only check in small time periods. I'll have to write a munin plugin
so I can get better monitoring on this.

I'm certain it is.

-Mathew Davies.

On 24 June 2011 11:27, Shay Banon shay.banon@elasticsearch.com wrote:

The rise in non heap memory usage is quite small (at least based on what
I can tell from the graph), and the heap usage is constant as well. Maybe
you answered it and I missed it, but are you sure its the ES process?

On Friday, June 24, 2011 at 1:09 PM, Mathew Davies wrote:

No.

On 24 June 2011 10:43, Shay Banon shay.banon@elasticsearch.com wrote:

Are you using scripts when you search?

On Thursday, June 23, 2011 at 9:20 PM, Mathew Davies wrote:

Elasticsearch memory usage still grows when very little is happening. I've
taken two new screenshots:

Old BigDesk : http://i.imgur.com/Xz21Y.jpg
New BigDesk: http://i.imgur.com/vseGU.jpg

The non-heap memory has expanded. Mem heap is around the same.

Old Munin: Imgur: The magic of the Internet
New Munin: Imgur: The magic of the Internet

Elasticsearch memory has slowly risen over the last few days to point
where's it's now eating into the swap again. All configuration is the same
as when I posted it further up.

Any help is greatly appreciated.

-Mathew Davies.

On 21 June 2011 15:14, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

14

This is still a problem, I stopped using the service wrapper as Shay
suggested in the IRC channel.

This is what munin looks like: http://i.imgur.com/bhnpI.jpg
and BigDesk: http://i.imgur.com/my4pT.jpg

There's memory being used somewhere which isn't being reported on bigdesk.

Total memory: 497M
Memory used by Elasticsearch: 36.2% (179.9M)

Bigdesk Heap (24M) + Non Heap (44M) = 68M

-Mathew Davies.

On 24 June 2011 12:17, Mathew Davies thepixeldeveloper@googlemail.comwrote:

The heap is now growing at a steady pace. In the first big-desk screenshot
you can see it topping it off at around 30MB (only 16 hours ago). It's
bordering on 50MB now.

Imgur: The magic of the Internet

I can't take a look at the usage over time because of the way big-desk
works and can only check in small time periods. I'll have to write a munin
plugin so I can get better monitoring on this.

I'm certain it is.

-Mathew Davies.

On 24 June 2011 11:27, Shay Banon shay.banon@elasticsearch.com wrote:

The rise in non heap memory usage is quite small (at least based on what
I can tell from the graph), and the heap usage is constant as well. Maybe
you answered it and I missed it, but are you sure its the ES process?

On Friday, June 24, 2011 at 1:09 PM, Mathew Davies wrote:

No.

On 24 June 2011 10:43, Shay Banon shay.banon@elasticsearch.com wrote:

Are you using scripts when you search?

On Thursday, June 23, 2011 at 9:20 PM, Mathew Davies wrote:

Elasticsearch memory usage still grows when very little is happening. I've
taken two new screenshots:

Old BigDesk : http://i.imgur.com/Xz21Y.jpg
New BigDesk: http://i.imgur.com/vseGU.jpg

The non-heap memory has expanded. Mem heap is around the same.

Old Munin: Imgur: The magic of the Internet
New Munin: Imgur: The magic of the Internet

Elasticsearch memory has slowly risen over the last few days to point
where's it's now eating into the swap again. All configuration is the same
as when I posted it further up.

Any help is greatly appreciated.

-Mathew Davies.

On 21 June 2011 15:14, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Matt

Here's a screenshot from bigdesk: http://i.imgur.com/Xz21Y.jpg

that looks pretty healthy to my (admittedly ignorant) eyes, but of
course it isn't running for long.

I'd be interested to see what it looks like when your machine starts
using lots of swap

clint

15

We are seeing the same issue. That is, running elasticsearch server
on linux with the sun jdk, the application memory seems constant, but
the swap memory steadily grows over the course of a week or so, until
we run out of swap space.

Additionally, I believe that our elastic client that runs in a Tomcat
webapp is leaking non-heap memory. After a period of time, the
process memory (but not the JVM memory) starts to grow steadily until
we have to restart the tomcat server. We never get a java "out of
memory" error, instead the java process eventually consumes all of
system memory.

We have been doing our best to isolate this, and it really does seem
to be the elastic client that is leaking the memory. Perhaps the
sever side swap memory growth is related.

I think we need help with this one.

Thanks,
Lar

On Jun 20, 11:25 am, Mathew Davies thepixeldevelo...@googlemail.com
wrote:

Any help appreciated diagnosing this problem, it's been bothering me
for days.

Here's a graph take from the last monthsmemoryusage:Imgur: The magic of the Internet
with annoations

  1. The server was restarted. Elasticsearch is running OpenJDK at this
    point.
  2. The appmemoryballoons and Elastic is restarted.
  3. Same as before.
  4. Switched to Suns JRE and now the appmemoryis under control, but
    the swap follows a very similar pattern to that of thememorywhen
    using OpenJDK

Elasticsearch configuration as from no. 4 on the graph.

bin/service/elasticsearch.conf -http://pastie.org/private/h3johakoudfe8m8wzojg
config/elasticsearch.yaml -http://pastie.org/private/aarhchbycpax8lwczjswa

Version: 0.16.2

Dataset

Around 3,000 entries with 4 string fields, 2 ints, 1 float and a date
time field. I would expect such a small data set to not use the amount
ofmemoryelastic is using.

System

Totalmemory= 469M
vm.swappiness = 10

Java

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)

Errors

  • This one is happening ever few hours: Jun 20 19:12:25 *** kernel:
    swapper: page allocation failure. order:5, mode:0x20
  • A frequent search term error I'm not too worried about, but could be
    a contributor:http://pastie.org/private/ceuoczbn65natwg6epm2a

If any more information is required then I'd be happy to help out.

Thanks for reading.

16

Try to find out which kind of memory the heap is piling up (Perma Old New). Tomcat is a notorious PermaGen leaker.

-------- Original Message --------

Subject: Re: Memory problems

From: lmader <lmaderintrepid@gmail.com>

Date: Wed, July 06, 2011 4:52 pm

To: users <users@elasticsearch.com>

We are seeing the same issue. That is, running elasticsearch server

on linux with the sun jdk, the application memory seems constant, but

the swap memory steadily grows over the course of a week or so, until

we run out of swap space.

Additionally, I believe that our elastic client that runs in a Tomcat

webapp is leaking non-heap memory. After a period of time, the

process memory (but not the JVM memory) starts to grow steadily until

we have to restart the tomcat server. We never get a java "out of

memory" error, instead the java process eventually consumes all of

system memory.

We have been doing our best to isolate this, and it really does seem

to be the elastic client that is leaking the memory. Perhaps the

sever side swap memory growth is related.

I think we need help with this one.

Thanks,

Lar

On Jun 20, 11:25 am, Mathew Davies <thepixeldevelo...@googlemail.com>

wrote:

> Any help appreciated diagnosing this problem, it's been bothering me

> for days.

>

> Here's a graph take from the last monthsmemoryusage:http://imgur.com/7FnJN

> with annoations

>

> 1. The server was restarted. Elasticsearch is running OpenJDK at this

> point.

> 2. The appmemoryballoons and Elastic is restarted.

> 3. Same as before.

> 4. Switched to Suns JRE and now the appmemoryis under control, but

> the swap follows a very similar pattern to that of thememorywhen

> using OpenJDK

>

> Elasticsearch configuration as from no. 4 on the graph.

> ----------

> bin/service/elasticsearch.conf -http://pastie.org/private/h3johakoudfe8m8wzojg

> config/elasticsearch.yaml -http://pastie.org/private/aarhchbycpax8lwczjswa

>

> Version: 0.16.2

>

> Dataset

> ----------

> Around 3,000 entries with 4 string fields, 2 ints, 1 float and a date

> time field. I would expect such a small data set to not use the amount

> ofmemoryelastic is using.

>

> System

> ----------

> Totalmemory= 469M

> vm.swappiness = 10

>

> Java

> ----------

> java version "1.6.0_24"

> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)

> Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)

>

> Errors

> ----------

> * This one is happening ever few hours: Jun 20 19:12:25 *** kernel:

> swapper: page allocation failure. order:5, mode:0x20

> * A frequent search term error I'm not too worried about, but could be

> a contributor:http://pastie.org/private/ceuoczbn65natwg6epm2a

>

> If any more information is required then I'd be happy to help out.

>

> Thanks for reading.

17

I'm not sure I follow your suggestion: As I mentioned, the memory
leak is not in the JVM memory - not in the regular heaps. The JVM
monitor shows normal memory usage and garbage collection patterns.
However the java process memory grows until all system memory is gone.

On Jul 6, 3:26 pm, jp.lora...@cfyar.com wrote:

Try to find out which kind ofmemorythe heap is piling up (Perma Old New). Tomcat is a notorious PermaGen leaker.

-------- Original Message --------
Subject: Re:Memoryproblems
From: lmader lmaderintrepid@gmail.com
Date: Wed, July 06, 2011 4:52 pm
To: users users@elasticsearch.com
We are seeing the same issue. That is, running elasticsearch server
on linux with the sun jdk, the applicationmemoryseems constant, but
the swapmemorysteadily grows over the course of a week or so, until
we run out of swap space.
Additionally, I believe that our elastic client that runs in a Tomcat
webapp is leaking non-heapmemory. After a period of time, the
processmemory(but not the JVMmemory) starts to grow steadily until
we have to restart the tomcat server. We never get a java "out ofmemory" error, instead the java process eventually consumes all of
systemmemory.
We have been doing our best to isolate this, and it really does seem
to be the elastic client that is leaking thememory. Perhaps the
sever side swapmemorygrowth is related.
I think we need help with this one.
Thanks,
Lar

18

Thats a difficult one the figure out. elasticsearch does not use native memory (by default), and this leak can come from allocations of native memory, bugs in native libraries used, or leaks in the java process itself. Are you using sigar within that app?

On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:

I'm not sure I follow your suggestion: As I mentioned, the memory
leak is not in the JVM memory - not in the regular heaps. The JVM
monitor shows normal memory usage and garbage collection patterns.
However the java process memory grows until all system memory is gone.

On Jul 6, 3:26 pm, <jp.lora...@cfyar.com (http://cfyar.com)> wrote:

Try to find out which kind ofmemorythe heap is piling up (Perma Old New). Tomcat is a notorious PermaGen leaker.

-------- Original Message --------
Subject: Re:Memoryproblems
From: lmader <lmaderintrepid@gmail.com (mailto:lmaderintrepid@gmail.com)>
Date: Wed, July 06, 2011 4:52 pm
To: users <users@elasticsearch.com (mailto:users@elasticsearch.com)>
We are seeing the same issue. That is, running elasticsearch server
on linux with the sun jdk, the applicationmemoryseems constant, but
the swapmemorysteadily grows over the course of a week or so, until
we run out of swap space.
Additionally, I believe that our elastic client that runs in a Tomcat
webapp is leaking non-heapmemory. After a period of time, the
processmemory(but not the JVMmemory) starts to grow steadily until
we have to restart the tomcat server. We never get a java "out ofmemory" error, instead the java process eventually consumes all of
systemmemory.
We have been doing our best to isolate this, and it really does seem
to be the elastic client that is leaking thememory. Perhaps the
sever side swapmemorygrowth is related.
I think we need help with this one.
Thanks,
Lar

19

We are not using sigar directly, unless ES does something with it. We
are running ES as a service.

Is it possible anything in the network layer or nio stuff could be
leaking?

Thanks so much,
Lar

On Jul 7, 7:20 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Thats a difficult one the figure out. elasticsearch does not use native memory (by default), and this leak can come from allocations of native memory, bugs in native libraries used, or leaks in the java process itself. Are you using sigar within that app?

On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:

I'm not sure I follow your suggestion: As I mentioned, the memory
leak is not in the JVM memory - not in the regular heaps. The JVM
monitor shows normal memory usage and garbage collection patterns.
However the java process memory grows until all system memory is gone.

On Jul 6, 3:26 pm, <jp.lora...@cfyar.com (http://cfyar.com)> wrote:

Try to find out which kind ofmemorythe heap is piling up (Perma Old New). Tomcat is a notorious PermaGen leaker.

-------- Original Message --------
Subject: Re:Memoryproblems
From: lmader <lmaderintre...@gmail.com (mailto:lmaderintre...@gmail.com)>
Date: Wed, July 06, 2011 4:52 pm
To: users <us...@elasticsearch.com (mailto:us...@elasticsearch.com)>
We are seeing the same issue. That is, running elasticsearch server
on linux with the sun jdk, the applicationmemoryseems constant, but
the swapmemorysteadily grows over the course of a week or so, until
we run out of swap space.
Additionally, I believe that our elastic client that runs in a Tomcat
webapp is leaking non-heapmemory. After a period of time, the
processmemory(but not the JVMmemory) starts to grow steadily until
we have to restart the tomcat server. We never get a java "out ofmemory" error, instead the java process eventually consumes all of
systemmemory.
We have been doing our best to isolate this, and it really does seem
to be the elastic client that is leaking thememory. Perhaps the
sever side swapmemorygrowth is related.
I think we need help with this one.
Thanks,
Lar

20

Its possible, I am trying to track this down and check.

On Friday, July 8, 2011 at 12:28 AM, lmader wrote:

We are not using sigar directly, unless ES does something with it. We
are running ES as a service.

Is it possible anything in the network layer or nio stuff could be
leaking?

Thanks so much,
Lar

On Jul 7, 7:20 am, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Thats a difficult one the figure out. elasticsearch does not use native memory (by default), and this leak can come from allocations of native memory, bugs in native libraries used, or leaks in the java process itself. Are you using sigar within that app?

On Thursday, July 7, 2011 at 2:17 AM, lmader wrote:

I'm not sure I follow your suggestion: As I mentioned, the memory
leak is not in the JVM memory - not in the regular heaps. The JVM
monitor shows normal memory usage and garbage collection patterns.
However the java process memory grows until all system memory is gone.

On Jul 6, 3:26 pm, <jp.lora...@cfyar.com (http://cfyar.com)> wrote:

Try to find out which kind ofmemorythe heap is piling up (Perma Old New). Tomcat is a notorious PermaGen leaker.

-------- Original Message --------
Subject: Re:Memoryproblems
From: lmader <lmaderintre...@gmail.com (mailto:lmaderintre...@gmail.com (http://gmail.com))>
Date: Wed, July 06, 2011 4:52 pm
To: users <us...@elasticsearch.com (mailto:us...@elasticsearch.com (http://elasticsearch.com))>
We are seeing the same issue. That is, running elasticsearch server
on linux with the sun jdk, the applicationmemoryseems constant, but
the swapmemorysteadily grows over the course of a week or so, until
we run out of swap space.
Additionally, I believe that our elastic client that runs in a Tomcat
webapp is leaking non-heapmemory. After a period of time, the
processmemory(but not the JVMmemory) starts to grow steadily until
we have to restart the tomcat server. We never get a java "out ofmemory" error, instead the java process eventually consumes all of
systemmemory.
We have been doing our best to isolate this, and it really does seem
to be the elastic client that is leaking thememory. Perhaps the
sever side swapmemorygrowth is related.
I think we need help with this one.
Thanks,
Lar