Suggested configuration to prevent OutOfMemoryError errors

Hi, I'm evaluating using ElasticSearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.

--
Richard Livsey
Minutebase - Online Meeting Minutes
http://minutebase.com
http://livsey.org

How do you index the data? If you push perform indexing in an async manner,
you should throttle it on the client side. In upcoming 0.10 I have improved
the blocking thread pool which is the one that you would want to use in this
case, but you might run into timeouts in this case.

-shay.banon

On Tue, Aug 24, 2010 at 7:04 PM, Richard Livsey livsey@gmail.com wrote:

Hi, I'm evaluating using Elasticsearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.

--
Richard Livsey
Minutebase - Online Meeting Minutes
http://minutebase.com
http://livsey.org

At the moment it's just looping through and indexing via the REST API,
which isn't async as it waits for timeouts (as far as I can tell).
But makes sense that it might be overloading it, so I'll add some
throttling and see how that goes.

Thanks.

On Tue, Aug 24, 2010 at 6:14 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

How do you index the data? If you push perform indexing in an async manner,
you should throttle it on the client side. In upcoming 0.10 I have improved
the blocking thread pool which is the one that you would want to use in this
case, but you might run into timeouts in this case.
-shay.banon

On Tue, Aug 24, 2010 at 7:04 PM, Richard Livsey livsey@gmail.com wrote:

Hi, I'm evaluating using Elasticsearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.

You are looping with a single thread and waiting for each index request? It
should not require much memory in this case... . What do you use to index
the data (I have a test with a single thread that indexed 1M docs with 128mb
allocated and it works like a charm).

-shay.banon

On Tue, Aug 24, 2010 at 8:21 PM, Richard Livsey richard@livsey.org wrote:

At the moment it's just looping through and indexing via the REST API,
which isn't async as it waits for timeouts (as far as I can tell).
But makes sense that it might be overloading it, so I'll add some
throttling and see how that goes.

Thanks.

On Tue, Aug 24, 2010 at 6:14 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

How do you index the data? If you push perform indexing in an async
manner,
you should throttle it on the client side. In upcoming 0.10 I have
improved
the blocking thread pool which is the one that you would want to use in
this
case, but you might run into timeouts in this case.
-shay.banon

On Tue, Aug 24, 2010 at 7:04 PM, Richard Livsey livsey@gmail.com
wrote:

Hi, I'm evaluating using Elasticsearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.