Failure to start elasticsearch as a service on t1.micro instances


(pulkitsinghal) #1

I've Google'd around a bit and I've been mostly been led to believe that I
shouldn't be using AWS t1.micro instances for elasticsearch as I'll run
into the "stale PID" problem. But what hasn't been made entirely clear to
me is: WHY the "stolen cpu" would even lead to such an issue for
elasticsearch?

  1. So, I was just mucking about on ssh when I strung together the following
    command w/ the shown results:
    $ sudo rm -rf /usr/local/var/run/elasticsearch/*.pid && sudo service
    elasticsearch start && sudo service elasticsearch status
    Starting elasticsearch...
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    initialize': A JSON text must at least contain two octets! (JSON::ParserError) from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:innew'
    from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    `parse'
    from -e:1
    elasticsearch running with PID 3528

  2. This is interesting because when I simply use the following I don't get
    any real info:
    $ sudo service elasticsearch start -f
    Starting elasticsearch...

  3. anyway even after running (1) I run into stale pid reported via status
    because clearly there has been an error that caused this process to end.
    That error being:
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    initialize': A JSON text must at least contain two octets! (JSON::ParserError) from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:innew'
    from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    `parse'
    from -e:1

So with these facts, my questions for the community are:

  1. What does this error indicate?
  2. How can it possibly be caused by the "stolen cpu" infrastructure problem
    that AWS t1.micro instances suffer from? Any guesses?
  3. Have you seen this error before in a completely different context?
  4. Is there a fix for this error?

Looking forward to everyone's remarks.

Cheers!

  • Pulkit

(Karel Minarik) #2

Hi,

ad 1/, this is specific to the service script at
[https://github.com/karmi/cookbook-elasticsearch]. The script is written
from the perspective "readable is better then robust", but this error
should indeed be catched better. There is a different service script at the
official
repo, https://github.com/elasticsearch/elasticsearch-servicewrapper.

The error is caused by the fact that when you do sudo service elasticsearch start immediately followed by sudo service elasticsearch status, ES not really up at that point, ant the script thus cannot load &
parse JSON information.

ad 2/, it is possible to run ES on EC2 micro, but you have to lower the
memory settings and use the engine "carefully". For most ES tests, micro is
a bad choice in my opinion, and running ES on micro in production would be
very challenging.

Karel

On Saturday, May 26, 2012 4:29:19 AM UTC+2, pulkitsinghal wrote:

I've Google'd around a bit and I've been mostly been led to believe that I
shouldn't be using AWS t1.micro instances for elasticsearch as I'll run
into the "stale PID" problem. But what hasn't been made entirely clear to
me is: WHY the "stolen cpu" would even lead to such an issue for
elasticsearch?

  1. So, I was just mucking about on ssh when I strung together the
    following command w/ the shown results:
    $ sudo rm -rf /usr/local/var/run/elasticsearch/*.pid && sudo service
    elasticsearch start && sudo service elasticsearch status
    Starting elasticsearch...
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    initialize': A JSON text must at least contain two octets! (JSON::ParserError) from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:innew'
    from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    `parse'
    from -e:1
    elasticsearch running with PID 3528

  2. This is interesting because when I simply use the following I don't get
    any real info:
    $ sudo service elasticsearch start -f
    Starting elasticsearch...

  3. anyway even after running (1) I run into stale pid reported via status
    because clearly there has been an error that caused this process to end.
    That error being:
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    initialize': A JSON text must at least contain two octets! (JSON::ParserError) from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:innew'
    from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    `parse'
    from -e:1

So with these facts, my questions for the community are:

  1. What does this error indicate?
  2. How can it possibly be caused by the "stolen cpu" infrastructure
    problem that AWS t1.micro instances suffer from? Any guesses?
  3. Have you seen this error before in a completely different context?
  4. Is there a fix for this error?

Looking forward to everyone's remarks.

Cheers!

  • Pulkit

(pulkitsinghal) #3

Hi Karel,

First of all, thank you for posting here in detail as well. I just saw your
reply on https://github.com/karmi/cookbook-elasticsearch/issues/11 and was
headed this way to correct & qualify my initial question as I realized
where that ruby code was really sitting. ES is java so pretty dumb of me
not to notice/suspect what was going on but such is life.

As for why I'm stuck on t1.micro? Well, its just such a pain to have to pay
for non-free (t1.micro is free for 1st year) instances like m1.large (yes,
i do forget to turn them off or my peers do too sometimes) while all I'm
doing is trying to come up with the perfect system to put into production
one day. Its sad because one snafu can ding me for $7 overnight. I suppose
I should go look into the vagrant approach of doing things but I feel AWS
is so touchy sometimes that I won't get it right later on, if I didn't
start with it in the 1st place.

Anyway, thanks ... you've helped me quite a bit :slight_smile:

Cheers,

  • Pulkit

On Saturday, May 26, 2012 12:45:55 AM UTC-5, Karel Minařík wrote:

Hi,

ad 1/, this is specific to the service script at [
https://github.com/karmi/cookbook-elasticsearch]. The script is written
from the perspective "readable is better then robust", but this error
should indeed be catched better. There is a different service script at the
official repo,
https://github.com/elasticsearch/elasticsearch-servicewrapper.

The error is caused by the fact that when you do sudo service elasticsearch start immediately followed by sudo service elasticsearch status, ES not really up at that point, ant the script thus cannot load &
parse JSON information.

ad 2/, it is possible to run ES on EC2 micro, but you have to lower the
memory settings and use the engine "carefully". For most ES tests, micro is
a bad choice in my opinion, and running ES on micro in production would be
very challenging.

Karel

On Saturday, May 26, 2012 4:29:19 AM UTC+2, pulkitsinghal wrote:

I've Google'd around a bit and I've been mostly been led to believe that
I shouldn't be using AWS t1.micro instances for elasticsearch as I'll run
into the "stale PID" problem. But what hasn't been made entirely clear to
me is: WHY the "stolen cpu" would even lead to such an issue for
elasticsearch?

  1. So, I was just mucking about on ssh when I strung together the
    following command w/ the shown results:
    $ sudo rm -rf /usr/local/var/run/elasticsearch/*.pid && sudo service
    elasticsearch start && sudo service elasticsearch status
    Starting elasticsearch...
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    initialize': A JSON text must at least contain two octets! (JSON::ParserError) from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:innew'
    from
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in `parse'
    from -e:1
    elasticsearch running with PID 3528

  2. This is interesting because when I simply use the following I don't
    get any real info:
    $ sudo service elasticsearch start -f
    Starting elasticsearch...

  3. anyway even after running (1) I run into stale pid reported via status
    because clearly there has been an error that caused this process to end.
    That error being:
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in
    initialize': A JSON text must at least contain two octets! (JSON::ParserError) from /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:innew'
    from
    /usr/lib/ruby/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in `parse'
    from -e:1

So with these facts, my questions for the community are:

  1. What does this error indicate?
  2. How can it possibly be caused by the "stolen cpu" infrastructure
    problem that AWS t1.micro instances suffer from? Any guesses?
  3. Have you seen this error before in a completely different context?
  4. Is there a fix for this error?

Looking forward to everyone's remarks.

Cheers!

  • Pulkit

(system) #4