Error in promotion to master?


(talsalmona) #1

Hi,

I have the following scenario (Windows Vista, ElasticSearch 0.8.0)

Configuration:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300"]

  1. Start node A --> It starts as master.
  2. Start node B.
  3. Stop A --> Node B prints that it has been promoted to master.
  4. Start node C.
    Expected behavior: Node C discovers node B as a master.
    Actual Behavior: Node C does not discover B and starts as a new master

Any idea why?
BTW, when I stop node B (A, the master keeps running) and start node C
then node C discovers A as the master.

Thanks,
Tal


(Shay Banon) #2

When you use unicast discovery, you need to list (as many as you can) nodes
that will form the cluster. In your case, you should list also
"localhost:9301" and "localhost:9302". Its ok if they are not up, as long as
a node that start up with an existing cluster has at least another node to
ask regarding the state of the cluster.

-shay.banon

On Sun, Jun 6, 2010 at 7:26 PM, Tal talsalmona@gmail.com wrote:

Hi,

I have the following scenario (Windows Vista, ElasticSearch 0.8.0)

Configuration:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300"]

  1. Start node A --> It starts as master.
  2. Start node B.
  3. Stop A --> Node B prints that it has been promoted to master.
  4. Start node C.
    Expected behavior: Node C discovers node B as a master.
    Actual Behavior: Node C does not discover B and starts as a new master

Any idea why?
BTW, when I stop node B (A, the master keeps running) and start node C
then node C discovers A as the master.

Thanks,
Tal


(talsalmona) #3

Got it.

Thanks,
Tal

On Jun 7, 3:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

When you use unicast discovery, you need to list (as many as you can) nodes
that will form the cluster. In your case, you should list also
"localhost:9301" and "localhost:9302". Its ok if they are not up, as long as
a node that start up with an existing cluster has at least another node to
ask regarding the state of the cluster.

-shay.banon

On Sun, Jun 6, 2010 at 7:26 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have the following scenario (Windows Vista, ElasticSearch 0.8.0)

Configuration:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300"]

  1. Start node A --> It starts as master.
  2. Start node B.
  3. Stop A --> Node B prints that it has been promoted to master.
  4. Start node C.
    Expected behavior: Node C discovers node B as a master.
    Actual Behavior: Node C does not discover B and starts as a new master

Any idea why?
BTW, when I stop node B (A, the master keeps running) and start node C
then node C discovers A as the master.

Thanks,
Tal


(Lenak) #4

Hi,

Another problem I've found is this...
After a client node was promoted to master, new nodes cannot connect
to it (they have the following exception):

[09:24:12,455][WARN ][discovery.zen ] [Oracle] Failed to
send join request to master [[Brand, Abigail][339bdebd-
e4ec-489b-9029-4b4c8eef6d66][inet[/16.59.79.252:9300]]], retrying...
org.elasticsearch.transport.RemoteTransportException: [Oracle][inet[/
16.59.79.252:9300]][discovery/zen/join]
Caused by: org.elasticsearch.ElasticSearchIllegalStateException: Node
[[Oracle][b371a93a-a0e6-44af-b291-78dc091e53a3][inet[/
16.59.79.252:9300]]{client=true, data=false, zen.master=false}] not
master for join request from [[Oracle][b371a93a-a0e6-44af- b291-78dc091e53a3][inet[/16.59.79.252:9300]]{client=true, data=false,
zen.master=false}]
at
org.elasticsearch.discovery.zen.ZenDiscovery.handleJoinRequest(ZenDiscovery.java:
386)
at org.elasticsearch.discovery.zen.ZenDiscovery.access
$900(ZenDiscovery.java:58)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MembershipListener.onJoin(ZenDiscovery.java:450)
at org.elasticsearch.discovery.zen.membership.MembershipAction
$JoinRequestRequestHandler.messageReceived(MembershipAction.java:113)
at org.elasticsearch.discovery.zen.membership.MembershipAction
$JoinRequestRequestHandler.messageReceived(MembershipAction.java:104)
at org.elasticsearch.transport.netty.MessageChannelHandler
$3.run(MessageChannelHandler.java:175)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

The configuration I used for the client node is this:

node:
data: false
client: true
discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300","localhost:9301","localhost:9302"]

and for the data node:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300","localhost:9301","localhost:9302"]

Using two data nodes every thing works great.
The reason for this might be that when I start a client node it does
not become a master and although in the scenario mentioned above the
client prints it has become a master it actually did not. I think a
client could become the master in previous versions.

Thanks,
Tal

On Jun 7, 5:29 pm, Tal talsalm...@gmail.com wrote:

Got it.

Thanks,
Tal

On Jun 7, 3:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

When you use unicast discovery, you need to list (as many as you can) nodes
that will form the cluster. In your case, you should list also
"localhost:9301" and "localhost:9302". Its ok if they are not up, as long as
a node that start up with an existing cluster has at least another node to
ask regarding the state of the cluster.

-shay.banon

On Sun, Jun 6, 2010 at 7:26 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have the following scenario (Windows Vista, ElasticSearch 0.8.0)

Configuration:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300"]

  1. Start node A --> It starts as master.
  2. Start node B.
  3. Stop A --> Node B prints that it has been promoted to master.
  4. Start node C.
    Expected behavior: Node C discovers node B as a master.
    Actual Behavior: Node C does not discover B and starts as a new master

Any idea why?
BTW, when I stop node B (A, the master keeps running) and start node C
then node C discovers A as the master.

Thanks,
Tal


(Shay Banon) #5

mmm, strange... . Let me have a look at this and report back.

-shay.banon

On Tue, Jun 8, 2010 at 9:43 AM, Lenak lenakovganov@gmail.com wrote:

Hi,

Another problem I've found is this...
After a client node was promoted to master, new nodes cannot connect
to it (they have the following exception):

[09:24:12,455][WARN ][discovery.zen ] [Oracle] Failed to
send join request to master [[Brand, Abigail][339bdebd-
e4ec-489b-9029-4b4c8eef6d66][inet[/16.59.79.252:9300]]], retrying...
org.elasticsearch.transport.RemoteTransportException: [Oracle][inet[/
16.59.79.252:9300]][discovery/zen/join]
Caused by: org.elasticsearch.ElasticSearchIllegalStateException: Node
[[Oracle][b371a93a-a0e6-44af-b291-78dc091e53a3][inet[/
16.59.79.252:9300]]{client=true, data=false, zen.master=false}] not
master for join request from [[Oracle][b371a93a-a0e6-44af- b291-78dc091e53a3][inet[/16.59.79.252:9300]]{client=true, data=false,
zen.master=false}]
at

org.elasticsearch.discovery.zen.ZenDiscovery.handleJoinRequest(ZenDiscovery.java:
386)
at org.elasticsearch.discovery.zen.ZenDiscovery.access
$900(ZenDiscovery.java:58)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MembershipListener.onJoin(ZenDiscovery.java:450)
at org.elasticsearch.discovery.zen.membership.MembershipAction
$JoinRequestRequestHandler.messageReceived(MembershipAction.java:113)
at org.elasticsearch.discovery.zen.membership.MembershipAction
$JoinRequestRequestHandler.messageReceived(MembershipAction.java:104)
at org.elasticsearch.transport.netty.MessageChannelHandler
$3.run(MessageChannelHandler.java:175)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

The configuration I used for the client node is this:

node:
data: false
client: true
discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300","localhost:9301","localhost:9302"]

and for the data node:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300","localhost:9301","localhost:9302"]

Using two data nodes every thing works great.
The reason for this might be that when I start a client node it does
not become a master and although in the scenario mentioned above the
client prints it has become a master it actually did not. I think a
client could become the master in previous versions.

Thanks,
Tal

On Jun 7, 5:29 pm, Tal talsalm...@gmail.com wrote:

Got it.

Thanks,
Tal

On Jun 7, 3:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

When you use unicast discovery, you need to list (as many as you can)
nodes

that will form the cluster. In your case, you should list also
"localhost:9301" and "localhost:9302". Its ok if they are not up, as
long as

a node that start up with an existing cluster has at least another node
to

ask regarding the state of the cluster.

-shay.banon

On Sun, Jun 6, 2010 at 7:26 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have the following scenario (Windows Vista, ElasticSearch 0.8.0)

Configuration:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300"]

  1. Start node A --> It starts as master.
  2. Start node B.
  3. Stop A --> Node B prints that it has been promoted to master.
  4. Start node C.
    Expected behavior: Node C discovers node B as a master.
    Actual Behavior: Node C does not discover B and starts as a new
    master

Any idea why?
BTW, when I stop node B (A, the master keeps running) and start node
C

then node C discovers A as the master.

Thanks,
Tal


(talsalmona) #6

This is fixed in 0.9.0. Client will never become a master.

Tal

On Jun 8, 12:20 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

mmm, strange... . Let me have a look at this and report back.

-shay.banon

On Tue, Jun 8, 2010 at 9:43 AM, Lenak lenakovga...@gmail.com wrote:

Hi,

Another problem I've found is this...
After a client node was promoted to master, new nodes cannot connect
to it (they have the following exception):

[09:24:12,455][WARN ][discovery.zen ] [Oracle] Failed to
send join request to master [[Brand, Abigail][339bdebd-
e4ec-489b-9029-4b4c8eef6d66][inet[/16.59.79.252:9300]]], retrying...
org.elasticsearch.transport.RemoteTransportException: [Oracle][inet[/
16.59.79.252:9300]][discovery/zen/join]
Caused by: org.elasticsearch.ElasticSearchIllegalStateException: Node
[[Oracle][b371a93a-a0e6-44af-b291-78dc091e53a3][inet[/
16.59.79.252:9300]]{client=true, data=false, zen.master=false}] not
master for join request from [[Oracle][b371a93a-a0e6-44af- b291-78dc091e53a3][inet[/16.59.79.252:9300]]{client=true, data=false,
zen.master=false}]
at

org.elasticsearch.discovery.zen.ZenDiscovery.handleJoinRequest(ZenDiscovery.java:
386)
at org.elasticsearch.discovery.zen.ZenDiscovery.access
$900(ZenDiscovery.java:58)
at org.elasticsearch.discovery.zen.ZenDiscovery
$MembershipListener.onJoin(ZenDiscovery.java:450)
at org.elasticsearch.discovery.zen.membership.MembershipAction
$JoinRequestRequestHandler.messageReceived(MembershipAction.java:113)
at org.elasticsearch.discovery.zen.membership.MembershipAction
$JoinRequestRequestHandler.messageReceived(MembershipAction.java:104)
at org.elasticsearch.transport.netty.MessageChannelHandler
$3.run(MessageChannelHandler.java:175)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

The configuration I used for the client node is this:

node:
data: false
client: true
discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300","localhost:9301","localhost:9302"]

and for the data node:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300","localhost:9301","localhost:9302"]

Using two data nodes every thing works great.
The reason for this might be that when I start a client node it does
not become a master and although in the scenario mentioned above the
client prints it has become a master it actually did not. I think a
client could become the master in previous versions.

Thanks,
Tal

On Jun 7, 5:29 pm, Tal talsalm...@gmail.com wrote:

Got it.

Thanks,
Tal

On Jun 7, 3:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

When you use unicast discovery, you need to list (as many as you can)
nodes

that will form the cluster. In your case, you should list also
"localhost:9301" and "localhost:9302". Its ok if they are not up, as
long as

a node that start up with an existing cluster has at least another node
to

ask regarding the state of the cluster.

-shay.banon

On Sun, Jun 6, 2010 at 7:26 PM, Tal talsalm...@gmail.com wrote:

Hi,

I have the following scenario (Windows Vista, ElasticSearch 0.8.0)

Configuration:

discovery.zen.ping.multicast:
enabled: false
discovery.zen.ping.unicast:
hosts: ["localhost:9300"]

  1. Start node A --> It starts as master.
  2. Start node B.
  3. Stop A --> Node B prints that it has been promoted to master.
  4. Start node C.
    Expected behavior: Node C discovers node B as a master.
    Actual Behavior: Node C does not discover B and starts as a new
    master

Any idea why?
BTW, when I stop node B (A, the master keeps running) and start node
C

then node C discovers A as the master.

Thanks,
Tal


(system) #7