ElasticSearch on EC2 -- trouble with the tutorial


(Rio) #1

Hello!

I'm trying to follow the tutorial (http://www.elasticsearch.org/
tutorials/2011/08/22/elasticsearch-on-ec2.html) to start an
Elasticsearch server on a EC2 instance, but can't seem to get the
server to start. I believe I'm following the instructions exactly.
Every time I try, CPU usage goes to 100% and the server stops
responding. I've tried 0.17.6 and 0.17.7 on micro basic 32bit and
64bit instances.

On the 32bit, I was getting a warning '', but found a irc log
directing me to start elasticsearch with the following command, which
seemed to resolve the warning message: ES_JAVA_OPTS='-server' bin/
elasticsearch -f

My configuration files are here: https://gist.github.com/1248557

I'm following these steps from the tutorial

  1. Launch up ec2 instance
  2. Download and install ES using these command

$ wget http://bit.ly/ES-0-17-6
$ sudo unzip elasticsearch-0.17.6.zip -d /usr/local/elasticsearch
$ cd /usr/local/elasticsearch/elasticsearch-0.17.6
$ sudo bin/plugin -install cloud-aws

  1. Configure ES https://gist.github.com/1248557
  2. Set environment variale ES_MAX_MEM=500m and EX_MIN_MEM=256m
    (roubly half of my instance memory)
  3. Start wtih sudo bin/elasticsearch -f

Any thoughts on what I'm doing wrong?

Thanks for all your help!


(Clinton Gormley) #2

Hiya

  1. Configure ES https://gist.github.com/1248557
  2. Set environment variale ES_MAX_MEM=500m and EX_MIN_MEM=256m
    (roubly half of my instance memory)

I assume you mean ES_MIN_MEM, not EX_MIN_MEM.

you should set ES_MIN_MEM to be the same as ES_MAX_MEM, for mlockall to
properly lock all the required memory

I think the problem is that a micro instance is just too small.

MIN/MAX_MEM refers to the heap size, not the total memory that ES uses.
The heap is usually the thing that can grow most, but the other memory
usage is significant, when you have so little memory to spare.

I think that ES is just running out of memory - I'd highly recommend
using a bigger instance

clint


(Rio) #3

Yep. You were right, the instance was just too small.

Thanks!

-Mario


(Karussell) #4

No, I don't think it is too small - it really depends on the amount of
data.

Just decrease the memory for MIN/MAX to 200 or 300MB and also decrease
the permgen size to 50.

On 28 Sep., 21:33, Rio rio...@gmail.com wrote:

Yep. You were right, the instance was just too small.

Thanks!

-Mario


(Jesús Roncero) #5

Hi,

I was also having difficulty with this trying to get it running (following
the tutorial) it on a micro instance on amazon. It can be run on that but
you have to reduce the memory to around 150 Mb (ES_MAX_MEM and ES_MIN_MEM).

To see if the OOM killer is running run dmesg and look for messages like:
[ 7938.981497] Out of memory: Kill process 22527 (java) score 894 or
sacrifice child

as the java process exits without any kind of message and/or error.

J.

On Thursday, 29 September 2011 08:23:49 UTC+1, Karussell wrote:

No, I don't think it is too small - it really depends on the amount of
data.

Just decrease the memory for MIN/MAX to 200 or 300MB and also decrease
the permgen size to 50.

On 28 Sep., 21:33, Rio rio...@gmail.com wrote:

Yep. You were right, the instance was just too small.

Thanks!

-Mario

--


(Derry O' Sullivan) #6

HI,

We have also had problems with java processes on micro instances (not just
ES), and it's a common problem. You can control the OOM though:

Also worth noting that a small instance seems to behave a lot better than a
micro (micro is just a burst usage instance) although it costs more (above
free usage tier).

On Thursday, 20 September 2012 14:31:32 UTC+1, Jesús Roncero wrote:

Hi,

I was also having difficulty with this trying to get it running (following
the tutorial) it on a micro instance on amazon. It can be run on that but
you have to reduce the memory to around 150 Mb (ES_MAX_MEM and ES_MIN_MEM).

To see if the OOM killer is running run dmesg and look for messages like:
[ 7938.981497] Out of memory: Kill process 22527 (java) score 894 or
sacrifice child

as the java process exits without any kind of message and/or error.

J.

On Thursday, 29 September 2011 08:23:49 UTC+1, Karussell wrote:

No, I don't think it is too small - it really depends on the amount of
data.

Just decrease the memory for MIN/MAX to 200 or 300MB and also decrease
the permgen size to 50.

On 28 Sep., 21:33, Rio rio...@gmail.com wrote:

Yep. You were right, the instance was just too small.

Thanks!

-Mario

--


(system) #7