Discuss the Elastic Stack

Very Strange Master Node Issue - Closed nodes not being removed

Elastic Stack Elasticsearch

ferhatsb (ferhatsb) April 3, 2013, 5:35pm 1

Hi All,

One of our clusters had an oom issue (v0.20.5) then we have restarted it.
Unfortunately our 2 node cluster become 3, master just adds new nodes, if
we restart one other node.

For instance _cluster/state returns;

{

"cluster_name" : "searchbox",

"master_node" : "CMP9rjMJTJ-iukNTR4Lmvg",

"blocks" : { },

"nodes" : {

"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"C9bJzhjlTLC9yUfkQOChhA" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"nJZlgVUlRpmm4tDTVXneTg" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}

},

and we get logs

[2013-04-03 17:26:35,928][WARN ][cluster.service ] [Hornet] failed
to reconnect to node
[searchbox1][Wg3iZM88QsanRtaQFYbiYg][inet[/10.147.179.29:9300]]{tag=level0,
master=true}

org.elasticsearch.transport.ConnectTransportException:
[searchbox1][inet[/10.147.179.29:9300]] connect_timeout[30s]

But when I get _cluser/nodes

{

"ok" : true,

"cluster_name" : "searchbox",

"nodes" : {

"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "hostname" : "ip-10-32-86-126",

  "version" : "0.20.5",

  "http_address" : "inet[/10.32.86.126:9200]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}

}

}

Any idea what is happening ?

Ferhat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ferhatsb (ferhatsb) April 3, 2013, 5:35pm 2

One addition we did not restart master.

On Wednesday, April 3, 2013 8:35:01 PM UTC+3, ferhatsb wrote:

Hi All,

One of our clusters had an oom issue (v0.20.5) then we have restarted it.
Unfortunately our 2 node cluster become 3, master just adds new nodes, if
we restart one other node.

For instance _cluster/state returns;

{

"cluster_name" : "searchbox",

"master_node" : "CMP9rjMJTJ-iukNTR4Lmvg",

"blocks" : { },

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"C9bJzhjlTLC9yUfkQOChhA" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"nJZlgVUlRpmm4tDTVXneTg" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
},

and we get logs

[2013-04-03 17:26:35,928][WARN ][cluster.service ] [Hornet]
failed to reconnect to node
[searchbox1][Wg3iZM88QsanRtaQFYbiYg][inet[/10.147.179.29:9300]]{tag=level0,
master=true}

org.elasticsearch.transport.ConnectTransportException: [searchbox1][inet[/
10.147.179.29:9300]] connect_timeout[30s]

But when I get _cluser/nodes

{

"ok" : true,

"cluster_name" : "searchbox",

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "hostname" : "ip-10-32-86-126",

  "version" : "0.20.5",

  "http_address" : "inet[/10.32.86.126:9200]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
}

}

Any idea what is happening ?

Ferhat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

simonw_2 (simonw-2) April 7, 2013, 10:52am 3

only one node hit the OOM or both?

simon

On Wednesday, April 3, 2013 7:35:01 PM UTC+2, ferhatsb wrote:

Hi All,

One of our clusters had an oom issue (v0.20.5) then we have restarted it.
Unfortunately our 2 node cluster become 3, master just adds new nodes, if
we restart one other node.

For instance _cluster/state returns;

{

"cluster_name" : "searchbox",

"master_node" : "CMP9rjMJTJ-iukNTR4Lmvg",

"blocks" : { },

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"C9bJzhjlTLC9yUfkQOChhA" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"nJZlgVUlRpmm4tDTVXneTg" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
},

and we get logs

[2013-04-03 17:26:35,928][WARN ][cluster.service ] [Hornet]
failed to reconnect to node
[searchbox1][Wg3iZM88QsanRtaQFYbiYg][inet[/10.147.179.29:9300]]{tag=level0,
master=true}

org.elasticsearch.transport.ConnectTransportException: [searchbox1][inet[/
10.147.179.29:9300]] connect_timeout[30s]

But when I get _cluser/nodes

{

"ok" : true,

"cluster_name" : "searchbox",

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "hostname" : "ip-10-32-86-126",

  "version" : "0.20.5",

  "http_address" : "inet[/10.32.86.126:9200]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
}

}

Any idea what is happening ?

Ferhat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ferhatsb (ferhatsb) April 8, 2013, 6:01am 4

Hi Simon,
Only one node, updated to v0.20.6 and restarted master node then problem
faded.
After 5 days I'm not sure if this is exactly Elasticsearch issue at all.
We have some customizations/extensions over rest api unfortunately I
couldn't verify the root cause.
I will update you with better information (thread dumps etc) if it happens
again.

On Sunday, April 7, 2013 1:52:16 PM UTC+3, simonw wrote:

only one node hit the OOM or both?

simon

On Wednesday, April 3, 2013 7:35:01 PM UTC+2, ferhatsb wrote:
Hi All,

One of our clusters had an oom issue (v0.20.5) then we have restarted it.
Unfortunately our 2 node cluster become 3, master just adds new nodes, if
we restart one other node.

For instance _cluster/state returns;

{

"cluster_name" : "searchbox",

"master_node" : "CMP9rjMJTJ-iukNTR4Lmvg",

"blocks" : { },

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"C9bJzhjlTLC9yUfkQOChhA" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"nJZlgVUlRpmm4tDTVXneTg" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
},

and we get logs

[2013-04-03 17:26:35,928][WARN ][cluster.service ] [Hornet]
failed to reconnect to node
[searchbox1][Wg3iZM88QsanRtaQFYbiYg][inet[/10.147.179.29:9300]]{tag=level0,
master=true}

org.elasticsearch.transport.ConnectTransportException: [searchbox1][inet[/
10.147.179.29:9300]] connect_timeout[30s]

But when I get _cluser/nodes

{

"ok" : true,

"cluster_name" : "searchbox",

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "hostname" : "ip-10-32-86-126",

  "version" : "0.20.5",

  "http_address" : "inet[/10.32.86.126:9200]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
}

}

Any idea what is happening ?

Ferhat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

simonw_2 (simonw-2) April 8, 2013, 6:33am 5

thanks!

On Monday, April 8, 2013 8:01:16 AM UTC+2, ferhatsb wrote:

Hi Simon,
Only one node, updated to v0.20.6 and restarted master node then problem
faded.
After 5 days I'm not sure if this is exactly Elasticsearch issue at all.
We have some customizations/extensions over rest api unfortunately I
couldn't verify the root cause.
I will update you with better information (thread dumps etc) if it happens
again.

On Sunday, April 7, 2013 1:52:16 PM UTC+3, simonw wrote:
only one node hit the OOM or both?

simon

On Wednesday, April 3, 2013 7:35:01 PM UTC+2, ferhatsb wrote:
Hi All,

One of our clusters had an oom issue (v0.20.5) then we have restarted
it. Unfortunately our 2 node cluster become 3, master just adds new nodes,
if we restart one other node.

For instance _cluster/state returns;

{

"cluster_name" : "searchbox",

"master_node" : "CMP9rjMJTJ-iukNTR4Lmvg",

"blocks" : { },

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"C9bJzhjlTLC9yUfkQOChhA" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

},

"nJZlgVUlRpmm4tDTVXneTg" : {

  "name" : "searchbox1",

  "transport_address" : "inet[/10.147.179.29:9300]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
},

and we get logs

[2013-04-03 17:26:35,928][WARN ][cluster.service ] [Hornet]
failed to reconnect to node
[searchbox1][Wg3iZM88QsanRtaQFYbiYg][inet[/10.147.179.29:9300]]{tag=level0,
master=true}

org.elasticsearch.transport.ConnectTransportException:
[searchbox1][inet[/10.147.179.29:9300]] connect_timeout[30s]

But when I get _cluser/nodes

{

"ok" : true,

"cluster_name" : "searchbox",

"nodes" : {
"CMP9rjMJTJ-iukNTR4Lmvg" : {

  "name" : "Hornet",

  "transport_address" : "inet[/10.32.86.126:9300]",

  "hostname" : "ip-10-32-86-126",

  "version" : "0.20.5",

  "http_address" : "inet[/10.32.86.126:9200]",

  "attributes" : {

    "tag" : "level0",

    "master" : "true"

  }

}
}

}

Any idea what is happening ?

Ferhat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views	Activity
Node not connected Elasticsearch	4	11833	July 6, 2017
Nodes continuously leaving and rejoining the cluster in 7.1 cluster after master switch Elasticsearch	8	1970	October 15, 2020
Master node discovery is not working Elasticsearch	2	609	February 28, 2018
Elasticsearch 6.1.3 -- failed to discover master after node restart Elasticsearch	6	1238	April 27, 2018
New Master not found Elasticsearch	1	284	July 6, 2017

© 2020. All Rights Reserved - Elasticsearch

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.