ElasticSearch 503 - ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]

Hi!

I'm using ES in 0.19.3 under ubuntu 12.10 64bit. After my system start i
got:

curl http://localhost:9200

{
"ok" : true,
"status" : 503,
"name" : "Selene",
"version" : {
"number" : "0.19.3",
"snapshot_build" : false
},
"tagline" : "You Know, for Search"

After the restart i got:

sudo /usr/local/share/elasticsearch/bin/service/elasticsearch condrestart
Stopping Elasticsearch...
Stopped Elasticsearch.
Starting Elasticsearch...
Waiting for Elasticsearch......
running: PID:6327

curl localhost:9200/_status
curl: (7) couldn't connect to host

In my logs it shows:

service.log:

STATUS | wrapper | 2013/01/24 11:35:41 | Launching a JVM...
INFO | jvm 1 | 2013/01/24 11:35:42 | WrapperManager: Initializing...
STATUS | wrapper | 2013/01/24 11:36:42 | TERM trapped. Shutting down.
STATUS | wrapper | 2013/01/24 11:36:43 | <-- Wrapper Stopped
STATUS | wrapper | 2013/01/24 11:36:44 | --> Wrapper Started as Daemon
STATUS | wrapper | 2013/01/24 11:36:44 | Java Service Wrapper Community
Edition 64-bit 3.5.14
STATUS | wrapper | 2013/01/24 11:36:44 | Copyright (C) 1999-2011 Tanuki
Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2013/01/24 11:36:44 | http:
//wrapper.tanukisoftware.com
STATUS | wrapper | 2013/01/24 11:36:44 |
STATUS | wrapper | 2013/01/24 11:36:44 | Launching a JVM...
INFO | jvm 1 | 2013/01/24 11:36:44 | WrapperManager: Initializing...

in the elasticsearch.log:

[2013-01-24 11:43:05,830][WARN ][transport.netty ] [Damballah]
Exception caught on netty layer [[id: 0x3827c87b]]
java.net.SocketException: Network is unreachable
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:99)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java
:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(
AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:553)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:503)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:482)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:128)
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(
ZenDiscovery.java:312)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.
java:69)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:
266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:603)
at java.lang.Thread.run(Thread.java:722)
[2013-01-24 11:43:05,830][WARN ][discovery.zen ] [Damballah]failed to connect to master
[[Mogul of the Mystic Mountain][6TALaMS6Q7eDPIUg8tZl4g][inet[/fe80:0:0:0:
3285:a9ff:fe42:388c%2:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Mogul of the Mystic
Mountain][inet[/fe80:0:0:0:3285:a9ff:fe42:388c%2:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:560)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:503)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:482)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:128)
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(
ZenDiscovery.java:312)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.
java:69)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:
266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Network is unreachable
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:99)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java
:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(
AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:547)
... 9 more

I'm not using a cluster just single node.

Anybody have similar problem?

Thx,
Max

--

I've make full uninstall of 0.19.3. Then install 0.20.2. And It started.

Thx,
Max

W dniu czwartek, 24 stycznia 2013 11:45:26 UTC+1 użytkownik Max Małecki
napisał:

Hi!

I'm using ES in 0.19.3 under ubuntu 12.10 64bit. After my system start i
got:

curl http://localhost:9200

{
"ok" : true,
"status" : 503,
"name" : "Selene",
"version" : {
"number" : "0.19.3",
"snapshot_build" : false
},
"tagline" : "You Know, for Search"

After the restart i got:

sudo /usr/local/share/elasticsearch/bin/service/elasticsearch condrestart
Stopping Elasticsearch...
Stopped Elasticsearch.
Starting Elasticsearch...
Waiting for Elasticsearch......
running: PID:6327

curl localhost:9200/_status
curl: (7) couldn't connect to host

In my logs it shows:

service.log:

STATUS | wrapper | 2013/01/24 11:35:41 | Launching a JVM...
INFO | jvm 1 | 2013/01/24 11:35:42 | WrapperManager: Initializing...
STATUS | wrapper | 2013/01/24 11:36:42 | TERM trapped. Shutting down.
STATUS | wrapper | 2013/01/24 11:36:43 | <-- Wrapper Stopped
STATUS | wrapper | 2013/01/24 11:36:44 | --> Wrapper Started as Daemon
STATUS | wrapper | 2013/01/24 11:36:44 | Java Service Wrapper Community
Edition 64-bit 3.5.14
STATUS | wrapper | 2013/01/24 11:36:44 | Copyright (C) 1999-2011 Tanuki
Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2013/01/24 11:36:44 | http://
wrapper.tanukisoftware.com
STATUS | wrapper | 2013/01/24 11:36:44 |
STATUS | wrapper | 2013/01/24 11:36:44 | Launching a JVM...
INFO | jvm 1 | 2013/01/24 11:36:44 | WrapperManager: Initializing...

in the elasticsearch.log:

[2013-01-24 11:43:05,830][WARN ][transport.netty ] [Damballah]
Exception caught on netty layer [[id: 0x3827c87b]]
java.net.SocketException: Network is unreachable
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:99)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.
java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(
AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:553)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:503)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:482)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:128)
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(
ZenDiscovery.java:312)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(
ZenDiscovery.java:69)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.
java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
[2013-01-24 11:43:05,830][WARN ][discovery.zen ] [Damballah]failed to connect to master
[[Mogul of the Mystic Mountain][6TALaMS6Q7eDPIUg8tZl4g][inet[/fe80:0:0:0:
3285:a9ff:fe42:388c%2:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Mogul of the
Mystic Mountain][inet[/fe80:0:0:0:3285:a9ff:fe42:388c%2:9300]]connect_timeout
[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:560)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:503)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
NettyTransport.java:482)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:128)
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(
ZenDiscovery.java:312)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(
ZenDiscovery.java:69)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.
java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Network is unreachable
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:99)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.
java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(
AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(
ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:547)
... 9 more

I'm not using a cluster just single node.

Anybody have similar problem?

Thx,
Max

--