Has anyone run into the trouble of starting up 2 ES instances? They both want to be masters on the same machine.
What's up with the "z" hostname that ES can't resolve? Where does it come from?
The reason they're both elected masters is because you're explicitly putting them in different clusters by passing the --cluster.name option.
Next time, please paste log output into the message instead of posting a screenshot.
z is the user name though I am just logged in as root. This was probably a bad example was testing the tribe node set up. But now here is what happens when I try to set the same cluster name or if I do not set a cluster name at all:
[root@z elasticsearch-1.5.1]# bin/elasticsearch -Des.node.data=false -Des.node.master=true -Des.node.name=NoData
[2015-06-25 14:30:25,357][INFO ][node ] [NoData] version[1.5.1], pid[6622], build[5e38401/2015-04-09T13:41:35Z]
[2015-06-25 14:30:25,357][INFO ][node ] [NoData] initializing ...
[2015-06-25 14:30:25,376][INFO ][plugins ] [NoData] loaded [], sites [head, kopf, bigdesk, paramedic, HQ]
[2015-06-25 14:30:29,478][WARN ][common.network ] failed to resolve local host, fallback to loopback
java.net.UnknownHostException: z: z: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1484)
at org.elasticsearch.common.network.NetworkUtils.<clinit>(NetworkUtils.java:55)
at org.elasticsearch.http.netty.NettyHttpServerTransport.<init>(NettyHttpServerTransport.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.net.UnknownHostException: z: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress.getLocalHost(InetAddress.java:1479)
... 29 more
[2015-06-25 14:30:29,846][INFO ][node ] [NoData] initialized
[2015-06-25 14:30:29,847][INFO ][node ] [NoData] starting ...
[2015-06-25 14:30:30,143][INFO ][transport ] [NoData] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.0.2.15:9300]}
[2015-06-25 14:30:30,252][INFO ][discovery ] [NoData] elasticsearch/3SwHjyNbR02vcopAGO8SWQ
[2015-06-25 14:30:34,040][INFO ][cluster.service ] [NoData] new_master [NoData][3SwHjyNbR02vcopAGO8SWQ][localhost][inet[/10.0.2.15:9300]]{data=false, master=true}, reason: zen-disco-join (elected_as_master)
[2015-06-25 14:30:34,123][INFO ][http ] [NoData] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.2.15:9200]}
[2015-06-25 14:30:34,123][INFO ][node ] [NoData] started
[2015-06-25 14:30:34,200][INFO ][gateway ] [NoData] recovered [8] indices into cluster_state
Then I set up another one:
[root@z elasticsearch-1.5.1]# bin/elasticsearch -Des.node.data=true -Des.node.master=false -Des.node.name=DataOne
[2015-06-25 14:25:09,321][INFO ][node ] [DataOne] version[1.5.1], pid[6534], build[5e38401/2015-04-09T13:41:35Z]
[2015-06-25 14:25:09,322][INFO ][node ] [DataOne] initializing ...
[2015-06-25 14:25:09,334][INFO ][plugins ] [DataOne] loaded [], sites []
[2015-06-25 14:25:13,338][WARN ][common.network ] failed to resolve local host, fallback to loopback
java.net.UnknownHostException: z: z: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1484)
at org.elasticsearch.common.network.NetworkUtils.<clinit>(NetworkUtils.java:55)
at org.elasticsearch.http.netty.NettyHttpServerTransport.<init>(NettyHttpServerTransport.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.net.UnknownHostException: z: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress.getLocalHost(InetAddress.java:1479)
... 29 more
[2015-06-25 14:25:13,468][INFO ][node ] [DataOne] initialized
[2015-06-25 14:25:13,480][INFO ][node ] [DataOne] starting ...
[2015-06-25 14:25:13,855][INFO ][transport ] [DataOne] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.0.2.15:9301]}
[2015-06-25 14:25:13,922][INFO ][discovery ] [DataOne] elasticsearch/K0FX7oZdTn-kqFryLSqXQw
[2015-06-25 14:25:43,922][WARN ][discovery ] [DataOne] waited for 30s and no initial state was set by the discovery
[2015-06-25 14:25:43,929][INFO ][http ] [DataOne] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.0.2.15:9201]}
[2015-06-25 14:25:43,933][INFO ][node ] [DataOne] started
I just took these arguments from stackoverflow hoping it would work.
I think I should try to figure out that z hostname that its trying to resolve
Yes, you really should figure out what the "z" hostname is about. But now your example doesn't show both nodes being elected master—which is obvious since you've disabled mastership for DataOne, which as of 14:25:43 is still looking for a master to bond with. How long did you wait after that? Wasn't it ever able to find the NoData node?
(I think ES beginners spend far too much time thinking about which nodes should be allowed to be masters. Unless you have a significant number of nodes, like double digits, dedicated master nodes are overkill. Because you'll want to have at least three master-eligible nodes you can easily end up with more master-only nodes than data nodes.)
Yeah that z host name is something strange I will figure that out I think I know whats wrong with that. (basically you just change it back to localhost)
Okay figured it out. I do not know why but the auto-detect of other nodes in ES was acting up. So I had to give it a basic config so they can all find each other. You mention creating multiple master nodes as overkill. Can you explain? I haven't had much time to mess around with my setup but right now I created a small cluster with 1 master node and 2 of them are set master: flase and 1 more set both master: false data: false (because I saw in a kibana set up page we should have load balancer... need to understand what that means).
An ES master node keep track of the cluster's nodes, indexes, shards etc. For small clusters this isn't a lot of work so a master-only machine will mostly be idle and a data node that's also a master won't be hit hard by having both roles.
With only a single master-eligible node, your cluster becomes inoperable if that node goes down. If you let all three nodes become masters and have one replica of each shard you can shut down any single node as long as the cluster starts out as green and things will continue to work.