Instant crash on startup


(mrblue) #1

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That´s it. Nothing in the log file and i can´t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i´m lost :frowning:

Maybe someone has seen this before ?


(Shay Banon) #2

Do you run it with elasticsearch -f? Run it with it and see what happens.
Also, see if the JVM crashed for some reason, you should see, from the work
directory of the process (where you ran it from) an hs_err file (or
something similar).

On Tue, Jan 3, 2012 at 8:02 PM, mrblue john.agricola.27@googlemail.comwrote:

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That´s it. Nothing in the log file and i can´t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i´m lost :frowning:

Maybe someone has seen this before ?


(mrblue) #3

Hi,

the above is the output from ./elasticsearch -f
No log file either in any elasticsearch dir, not under /tmp , nothing
in tha java dir :frowning:
Hetzner server support will suffer tomorrow :wink:

On Jan 3, 7:50 pm, Shay Banon kim...@gmail.com wrote:

Do you run it with elasticsearch -f? Run it with it and see what happens.
Also, see if the JVM crashed for some reason, you should see, from the work
directory of the process (where you ran it from) an hs_err file (or
something similar).

On Tue, Jan 3, 2012 at 8:02 PM, mrblue john.agricola...@googlemail.comwrote:

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That´s it. Nothing in the log file and i can´t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i´m lost :frowning:

Maybe someone has seen this before ?


(Tomasz Kloc) #4

out of memory? out of disk space? any useful information in dmesg?
(lucky guessing)

On 03.01.2012 20:11, mrblue wrote:

Hi,

the above is the output from ./elasticsearch -f
No log file either in any elasticsearch dir, not under /tmp , nothing
in tha java dir :frowning:
Hetzner server support will suffer tomorrow :wink:

On Jan 3, 7:50 pm, Shay Banonkim...@gmail.com wrote:

Do you run it with elasticsearch -f? Run it with it and see what happens.
Also, see if the JVM crashed for some reason, you should see, from the work
directory of the process (where you ran it from) an hs_err file (or
something similar).

On Tue, Jan 3, 2012 at 8:02 PM, mrbluejohn.agricola...@googlemail.comwrote:

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That´s it. Nothing in the log file and i can´t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i´m lost :frowning:

Maybe someone has seen this before ?


(mrblue) #5

Nah, nothing. Big managed Server.. Tried it on a similar machine from
the same hoster, same problem. Will bother hoster tomorrow. Really
sucks.

On Jan 3, 8:15 pm, Tomasz Kloc tomek.kloc....@gmail.com wrote:

out of memory? out of disk space? any useful information in dmesg?
(lucky guessing)

On 03.01.2012 20:11, mrblue wrote:

Hi,

the above is the output from ./elasticsearch -f
No log file either in any elasticsearch dir, not under /tmp , nothing
in tha java dir :frowning:
Hetzner server support will suffer tomorrow :wink:

On Jan 3, 7:50 pm, Shay Banonkim...@gmail.com wrote:

Do you run it with elasticsearch -f? Run it with it and see what happens.
Also, see if the JVM crashed for some reason, you should see, from the work
directory of the process (where you ran it from) an hs_err file (or
something similar).

On Tue, Jan 3, 2012 at 8:02 PM, mrbluejohn.agricola...@googlemail.comwrote:

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That�s it. Nothing in the log file and i can�t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i�m lost :frowning:

Maybe someone has seen this before ?


(Tomasz Kloc) #6

So, my last guess is some watchdog-script kills all processes which
listen on 'blacklisted' port.

and update:
"""

  • http://www.hetzner.de/en/hosting/legal/system-policies-ws#6Process
    Monitoring:*
    To enable fair resource allocation there is a process monitoring system
    on shared servers which stops processes with excessive running times
    and/or RAM exploitation. For account types from Level 19 exceptions can
    be made for specifically defined processes, after they are first tested
    by the system administration. If the stability of the server or the
    network is affected, Hetzner Online has the right to correct and/or
    cancel these exceptions.
    """
    http://www.hetzner.de/en/hosting/legal/system-policies-ws

On 03.01.2012 20:27, mrblue wrote:

Nah, nothing. Big managed Server.. Tried it on a similar machine from
the same hoster, same problem. Will bother hoster tomorrow. Really
sucks.

On Jan 3, 8:15 pm, Tomasz Kloctomek.kloc....@gmail.com wrote:

out of memory? out of disk space? any useful information in dmesg?
(lucky guessing)

On 03.01.2012 20:11, mrblue wrote:

Hi,

the above is the output from ./elasticsearch -f
No log file either in any elasticsearch dir, not under /tmp , nothing
in tha java dir :frowning:
Hetzner server support will suffer tomorrow :wink:

On Jan 3, 7:50 pm, Shay Banonkim...@gmail.com wrote:

Do you run it with elasticsearch -f? Run it with it and see what happens.
Also, see if the JVM crashed for some reason, you should see, from the work
directory of the process (where you ran it from) an hs_err file (or
something similar).

On Tue, Jan 3, 2012 at 8:02 PM, mrbluejohn.agricola...@googlemail.comwrote:

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That�s it. Nothing in the log file and i can�t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i�m lost :frowning:

Maybe someone has seen this before ?


(mrblue) #7

I´ll have a look at this, thx.
..but i don´t have a shared server, it´s a managed server for 120€/
month :frowning:

On Jan 3, 8:34 pm, Tomasz Kloc tomek.kloc....@gmail.com wrote:

So, my last guess is some watchdog-script kills all processes which
listen on 'blacklisted' port.

and update:
"""

  • http://www.hetzner.de/en/hosting/legal/system-policies-ws#6Process
    Monitoring:*
    To enable fair resource allocation there is a process monitoring system
    on shared servers which stops processes with excessive running times
    and/or RAM exploitation. For account types from Level 19 exceptions can
    be made for specifically defined processes, after they are first tested
    by the system administration. If the stability of the server or the
    network is affected, Hetzner Online has the right to correct and/or
    cancel these exceptions.
    """http://www.hetzner.de/en/hosting/legal/system-policies-ws

On 03.01.2012 20:27, mrblue wrote:

Nah, nothing. Big managed Server.. Tried it on a similar machine from
the same hoster, same problem. Will bother hoster tomorrow. Really
sucks.

On Jan 3, 8:15 pm, Tomasz Kloctomek.kloc....@gmail.com wrote:

out of memory? out of disk space? any useful information in dmesg?
(lucky guessing)

On 03.01.2012 20:11, mrblue wrote:

Hi,

the above is the output from ./elasticsearch -f
No log file either in any elasticsearch dir, not under /tmp , nothing
in tha java dir :frowning:
Hetzner server support will suffer tomorrow :wink:

On Jan 3, 7:50 pm, Shay Banonkim...@gmail.com wrote:

Do you run it with elasticsearch -f? Run it with it and see what happens.
Also, see if the JVM crashed for some reason, you should see, from the work
directory of the process (where you ran it from) an hs_err file (or
something similar).

On Tue, Jan 3, 2012 at 8:02 PM, mrbluejohn.agricola...@googlemail.comwrote:

Hi,

when i try to start elasticsearch on my managed server, it goes like
this:

[2012-01-03 18:59:25,364][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initializing ...
[2012-01-03 18:59:25,373][INFO ][plugins ] [Tom
Thumb] loaded [], sites []
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: initialized
[2012-01-03 18:59:27,788][INFO ][node ] [Tom
Thumb] {0.18.6}[23920]: starting ...
Killed

That�s it. Nothing in the log file and i can�t find the java log.
I had to set discovery.zen.ping.multicast.enabled: false in order to
work earlier, after a server restart i�m lost :frowning:

Maybe someone has seen this before ?


(Drew Raines) #8

mrblue wrote:

I´ll have a look at this, thx.
..but i don´t have a shared server, it´s a managed server for 120€/
month :frowning:

It doesn't quite fit the behavior, but check if there are oom-killer
messages in your kernel log (assuming this is Linux).

You could also try strace/ktrace/dtrace-ing the JVM process to see
if that reveals any clues.

-Drew


(mrblue) #9

Oh well.. Hetzner is limiting the RAM for java aplications to 120MB
default. They made it 300 for me, but that didn´t work too. (Seems
strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need ?

I probably will try a cheap root server, wondering if 8gig RAM are
enough ?

thx for your help, great community here

On Jan 3, 9:35 pm, Drew Raines aarai...@gmail.com wrote:

mrblue wrote:

I´ll have a look at this, thx.
..but i don´t have a shared server, it´s a managed server for 120€/
month :frowning:

It doesn't quite fit the behavior, but check if there are oom-killer
messages in your kernel log (assuming this is Linux).

You could also try strace/ktrace/dtrace-ing the JVM process to see
if that reveals any clues.

-Drew


(Drew Raines) #10

mrblue wrote:

Oh well.. Hetzner is limiting the RAM for java aplications to 120MB
default. They made it 300 for me, but that didn´t work too. (Seems
strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need ?

Totally depends on your data. Best thing to do is profile exactly
what you want to do. It only takes a few seconds to index that many
docs.

-Drew


(James Chisholm) #11

If it's not logging anything... does the user that elasticsearch is
running at have write permissions to the log folder?

On Jan 5, 8:07 am, Drew Raines aarai...@gmail.com wrote:

mrblue wrote:

Oh well.. Hetzner is limiting the RAM for java aplications to 120MB
default. They made it 300 for me, but that didn´t work too. (Seems
strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need ?

Totally depends on your data. Best thing to do is profile exactly
what you want to do. It only takes a few seconds to index that many
docs.

-Drew


(mrblue) #12

I can access /var/log but not every file there + there is no file that
"looks" like the right one.

On Jan 5, 4:03 am, James Chisholm ja...@someones.com wrote:

If it's not logging anything... does the user that elasticsearch is
running at have write permissions to the log folder?

On Jan 5, 8:07 am, Drew Raines aarai...@gmail.com wrote:

mrblue wrote:

Oh well.. Hetzner is limiting the RAM for java aplications to 120MB
default. They made it 300 for me, but that didn´t work too. (Seems
strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need ?

Totally depends on your data. Best thing to do is profile exactly
what you want to do. It only takes a few seconds to index that many
docs.

-Drew


(mrblue) #13

Allowing the JVM to use more RAM did the trick :slight_smile:

On Jan 5, 9:29 am, mrblue john.agricola...@googlemail.com wrote:

I can access /var/log but not every file there + there is no file that
"looks" like the right one.

On Jan 5, 4:03 am, James Chisholm ja...@someones.com wrote:

If it's not logging anything... does the user that elasticsearch is
running at have write permissions to the log folder?

On Jan 5, 8:07 am, Drew Raines aarai...@gmail.com wrote:

mrblue wrote:

Oh well.. Hetzner is limiting the RAM for java aplications to 120MB
default. They made it 300 for me, but that didn´t work too. (Seems
strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need ?

Totally depends on your data. Best thing to do is profile exactly
what you want to do. It only takes a few seconds to index that many
docs.

-Drew


(hazzadous) #14

I'm having issues getting things running on Hetzner too. Although not
exactly the same issues as you had (my symptoms are
https://groups.google.com/d/topic/elasticsearch/uqLCOYjit-o/discussion)
I'd be interested to know what setup you have.

On Thursday, January 5, 2012 2:48:13 PM UTC, mrblue wrote:

Allowing the JVM to use more RAM did the trick :slight_smile:

On Jan 5, 9:29 am, mrblue john.agricola...@googlemail.com wrote:

I can access /var/log but not every file there + there is no file that
"looks" like the right one.

On Jan 5, 4:03 am, James Chisholm ja...@someones.com wrote:

If it's not logging anything... does the user that elasticsearch is
running at have write permissions to the log folder?

On Jan 5, 8:07 am, Drew Raines aarai...@gmail.com wrote:

mrblue wrote:

Oh well.. Hetzner is limiting the RAM for java aplications to
120MB

default. They made it 300 for me, but that didn´t work too. (Seems
strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need ?

Totally depends on your data. Best thing to do is profile exactly
what you want to do. It only takes a few seconds to index that many
docs.

-Drew


(hazzadous) #15

The issue for me was the dreaded leap second issue, all smiles now.

On Wednesday, July 4, 2012 6:26:09 PM UTC+1, hazzadous wrote:

I'm having issues getting things running on Hetzner too. Although not
exactly the same issues as you had (my symptoms are
https://groups.google.com/d/topic/elasticsearch/uqLCOYjit-o/discussion)
I'd be interested to know what setup you have.

On Thursday, January 5, 2012 2:48:13 PM UTC, mrblue wrote:

Allowing the JVM to use more RAM did the trick :slight_smile:

On Jan 5, 9:29 am, mrblue john.agricola...@googlemail.com wrote:

I can access /var/log but not every file there + there is no file that
"looks" like the right one.

On Jan 5, 4:03 am, James Chisholm ja...@someones.com wrote:

If it's not logging anything... does the user that elasticsearch is
running at have write permissions to the log folder?

On Jan 5, 8:07 am, Drew Raines aarai...@gmail.com wrote:

mrblue wrote:

Oh well.. Hetzner is limiting the RAM for java aplications to
120MB

default. They made it 300 for me, but that didn´t work too.
(Seems

strange but nothing i can do..)

Does anyone know how much a small index (<10k entrys) would need
?

Totally depends on your data. Best thing to do is profile exactly
what you want to do. It only takes a few seconds to index that
many

docs.

-Drew


(system) #16