Suggested configuration to prevent OutOfMemoryError errors


(Richard Livsey) #1

Hi, I'm evaluating using ElasticSearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.

--
Richard Livsey
Minutebase - Online Meeting Minutes
http://minutebase.com
http://livsey.org


(Shay Banon) #2

How do you index the data? If you push perform indexing in an async manner,
you should throttle it on the client side. In upcoming 0.10 I have improved
the blocking thread pool which is the one that you would want to use in this
case, but you might run into timeouts in this case.

-shay.banon

On Tue, Aug 24, 2010 at 7:04 PM, Richard Livsey livsey@gmail.com wrote:

Hi, I'm evaluating using ElasticSearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.

--
Richard Livsey
Minutebase - Online Meeting Minutes
http://minutebase.com
http://livsey.org


(Richard Livsey-2) #3

At the moment it's just looping through and indexing via the REST API,
which isn't async as it waits for timeouts (as far as I can tell).
But makes sense that it might be overloading it, so I'll add some
throttling and see how that goes.

Thanks.

On Tue, Aug 24, 2010 at 6:14 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

How do you index the data? If you push perform indexing in an async manner,
you should throttle it on the client side. In upcoming 0.10 I have improved
the blocking thread pool which is the one that you would want to use in this
case, but you might run into timeouts in this case.
-shay.banon

On Tue, Aug 24, 2010 at 7:04 PM, Richard Livsey livsey@gmail.com wrote:

Hi, I'm evaluating using ElasticSearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.


(Shay Banon) #4

You are looping with a single thread and waiting for each index request? It
should not require much memory in this case... . What do you use to index
the data (I have a test with a single thread that indexed 1M docs with 128mb
allocated and it works like a charm).

-shay.banon

On Tue, Aug 24, 2010 at 8:21 PM, Richard Livsey richard@livsey.org wrote:

At the moment it's just looping through and indexing via the REST API,
which isn't async as it waits for timeouts (as far as I can tell).
But makes sense that it might be overloading it, so I'll add some
throttling and see how that goes.

Thanks.

On Tue, Aug 24, 2010 at 6:14 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

How do you index the data? If you push perform indexing in an async
manner,
you should throttle it on the client side. In upcoming 0.10 I have
improved
the blocking thread pool which is the one that you would want to use in
this
case, but you might run into timeouts in this case.
-shay.banon

On Tue, Aug 24, 2010 at 7:04 PM, Richard Livsey livsey@gmail.com
wrote:

Hi, I'm evaluating using ElasticSearch and just trying to get it up
and running on my laptop (MacBook Pro running Snow Leopard) but keep
running into java.lang.OutOfMemoryError errors.

I don't have much free RAM on this machine, what's the best way to
configure ES to run in a low memory environment without dying?

I've set ES_MAX_MEM as 512m, and set the threadpool to "scaling" with
the following settings:

"threadpool": {
"type": "scaling",
"scaling": {
"min": 1,
"max": 5,
"scheduled_size": 5
}
}

I'm only indexing a few hundred documents, some with files upto 20Mb
to be indexed as attachments.

Prior to lowering ES_MAX_MEM, it wouldn't throw the exception but
would freeze up as it GC'd:

[16:19:59,281][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [16.1s], breached threshold [10s]
[16:20:18,901][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [17.6s], breached threshold [10s]
[16:20:30,653][WARN ][monitor.jvm ] [Slapstick] Long GC
collection occurred, took [10.9s], breached threshold [10s]

Thanks.


(system) #5