0.90.2 Memory bug


(Alexis Okuwa) #1

Having a werid issue where elasticsearch is using a lot of memory. I have
the jvm settings to not use more then 1g but its using about 7gb of memory.
Linux is reporting it using about 1g as it should be. But when i kill the
proc all the memory gets flushed.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Martijn Van Groningen) #2

Where do you see that 7GB is used, node stats api? The jvm setting
ES_HEAP_SIZE only sets the jvm heapspace which is just one part of the
total memory that ES uses.

On 18 September 2013 11:57, Alexis Okuwa wojonstech@gmail.com wrote:

Having a werid issue where elasticsearch is using a lot of memory. I have
the jvm settings to not use more then 1g but its using about 7gb of memory.
Linux is reporting it using about 1g as it should be. But when i kill the
proc all the memory gets flushed.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #3

when i look at top or htop it shows 7gb being used and if i kill
elasticsearch it that memory usage goes away.

On Wed, Sep 18, 2013 at 3:07 AM, Martijn v Groningen <
martijn.v.groningen@gmail.com> wrote:

Where do you see that 7GB is used, node stats api? The jvm setting
ES_HEAP_SIZE only sets the jvm heapspace which is just one part of the
total memory that ES uses.

On 18 September 2013 11:57, Alexis Okuwa wojonstech@gmail.com wrote:

Having a werid issue where elasticsearch is using a lot of memory. I have
the jvm settings to not use more then 1g but its using about 7gb of memory.
Linux is reporting it using about 1g as it should be. But when i kill the
proc all the memory gets flushed.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #4

What JVM settings did you use?

If you did not set ES_DIRECT_SIZE, the memory usage for direct buffers is
unlimited by default. That means the JVM lends memory from the OS until the
OS does not have any more. It's not a bug, it's a feature to exploit the OS
for best performance and resource usage.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #5

Well its using 4gb of ram, and 3+gb of swap.

On Thu, Sep 19, 2013 at 12:20 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

What JVM settings did you use?

If you did not set ES_DIRECT_SIZE, the memory usage for direct buffers is
unlimited by default. That means the JVM lends memory from the OS until the
OS does not have any more. It's not a bug, it's a feature to exploit the OS
for best performance and resource usage.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #6

So i set the value that you pointed out to 2g. and i have Xmx at 1gb but i
am still using about 3.5gb of really memory and 999mb of swap. but htop is
reporting that its only using 17.1% of memory

On Thu, Sep 19, 2013 at 3:03 PM, Alexis Okuwa wojonstech@gmail.com wrote:

Well its using 4gb of ram, and 3+gb of swap.

On Thu, Sep 19, 2013 at 12:20 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

What JVM settings did you use?

If you did not set ES_DIRECT_SIZE, the memory usage for direct buffers is
unlimited by default. That means the JVM lends memory from the OS until the
OS does not have any more. It's not a bug, it's a feature to exploit the OS
for best performance and resource usage.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #7

It is impossible to comment without seeing all the ES configs and the
output values of your diagnostic programs.

Maybe you run 32bit JVM and can't lock ES via mmap into RAM.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #8

This is the version i am downloading using my configuration managment
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.2.deb

What config files do you need?

/usr/share/elasticsearch/bin/elasticsearch -v

ElasticSearch Version: 0.90.2, JVM: 1.7.0_25(23.25-b01)

On Sat, Sep 21, 2013 at 3:05 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

It is impossible to comment without seeing all the ES configs and the
output values of your diagnostic programs.

Maybe you run 32bit JVM and can't lock ES via mmap into RAM.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #9

Still wondering about your machine, what OS and if JVM is 32/64 bit.

If you have only 4g RAM total, and OS and file system cache take their
space, then you have much less than 2g heap for ES, maybe just 1g which is
the standard config.

I suggest 32bit JVM and running with standard config.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #10

java -version

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

/usr/lib/jvm/java-7-oracle/bin/java -Xms256m -Xmx1g
-XX:MaxDirectMemorySize=2g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.pidfile=/var/run/elasticsearch.pid
-Des.path.home=/usr/share/elasticsearch -cp
:/usr/share/elasticsearch/lib/elasticsearch-0.90.2.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.config=/etc/elasticsearch/elasticsearch.yml
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.ElasticSearch

Linux myhost 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux

On Sun, Sep 22, 2013 at 2:17 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Still wondering about your machine, what OS and if JVM is 32/64 bit.

If you have only 4g RAM total, and OS and file system cache take their
space, then you have much less than 2g heap for ES, maybe just 1g which is
the standard config.

I suggest 32bit JVM and running with standard config.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #11

ping?

On Sun, Sep 22, 2013 at 4:17 AM, Alexis Okuwa wojonstech@gmail.com wrote:

java -version

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

/usr/lib/jvm/java-7-oracle/bin/java -Xms256m -Xmx1g
-XX:MaxDirectMemorySize=2g -Xss256k -Djava.awt.headless=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch
-Des.pidfile=/var/run/elasticsearch.pid
-Des.path.home=/usr/share/elasticsearch -cp
:/usr/share/elasticsearch/lib/elasticsearch-0.90.2.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
-Des.default.config=/etc/elasticsearch/elasticsearch.yml
-Des.default.path.home=/usr/share/elasticsearch
-Des.default.path.logs=/var/log/elasticsearch
-Des.default.path.data=/var/lib/elasticsearch
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/etc/elasticsearch
org.elasticsearch.bootstrap.ElasticSearch

Linux myhost 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux

On Sun, Sep 22, 2013 at 2:17 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Still wondering about your machine, what OS and if JVM is 32/64 bit.

If you have only 4g RAM total, and OS and file system cache take their
space, then you have much less than 2g heap for ES, maybe just 1g which is
the standard config.

I suggest 32bit JVM and running with standard config.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #12

Your settings seem too high for 4g total RAM. You allocate Java Program
Binary (~100m) + Stack (~100m) + Heap (1g) + direct buffers (2g) which
alone is ~3,5g and is far more than recommended 4g / 2 = 2g, so it is not
surprisingly using swap memory.

Please do not set direct buffer to 2g, this makes no sense on 4g total RAM
machine.

Also, you use 64bit Java, which uses more memory than 32bit Java. Note that
32bit Java does not make use of direct buffers. Did you try 32bit Java by
using option -d32 ?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #13

I can increases the ram, the vm is inside of a dedicated server that I own.
I been running this machine for months and never had a memory issue so its
just very werid that it is coming up now.

On Tue, Sep 24, 2013 at 12:22 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Your settings seem too high for 4g total RAM. You allocate Java Program
Binary (~100m) + Stack (~100m) + Heap (1g) + direct buffers (2g) which
alone is ~3,5g and is far more than recommended 4g / 2 = 2g, so it is not
surprisingly using swap memory.

Please do not set direct buffer to 2g, this makes no sense on 4g total RAM
machine.

Also, you use 64bit Java, which uses more memory than 32bit Java. Note
that 32bit Java does not make use of direct buffers. Did you try 32bit Java
by using option -d32 ?

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexis Okuwa) #14

I have seen elasticsearch now go up to 7.5gb of ram+swap usage based on my
settings elasticserch seems to be ignoring them.

On Tue, Sep 24, 2013 at 12:36 AM, Alexis Okuwa wojonstech@gmail.com wrote:

I can increases the ram, the vm is inside of a dedicated server that I
own. I been running this machine for months and never had a memory issue so
its just very werid that it is coming up now.

On Tue, Sep 24, 2013 at 12:22 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Your settings seem too high for 4g total RAM. You allocate Java Program
Binary (~100m) + Stack (~100m) + Heap (1g) + direct buffers (2g) which
alone is ~3,5g and is far more than recommended 4g / 2 = 2g, so it is not
surprisingly using swap memory.

Please do not set direct buffer to 2g, this makes no sense on 4g total
RAM machine.

Also, you use 64bit Java, which uses more memory than 32bit Java. Note
that 32bit Java does not make use of direct buffers. Did you try 32bit Java
by using option -d32 ?

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qfPiUVgJ5K0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
Enjoy,
Alexis Okuwa
WojonsTech
424.835.1223

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #15