"netstat -lntup | grep 9" shows all java processes.
ps -ef | grep elastic correctly shows the elasticsearch process in the directory I had used in the config and plugin.
I guess tribe.t2.path.conf: and tribe.t2.path.plugins: should point to the folders in the tribe node and not in individual master nodes of t1 and t2. Is that correct understanding?
If you shutdown the tribe node, and do the netstat again, is port 9300 used by any processes?
Tried this. Nothing is running in 9300.
[root@cheupiapp04 ~]# netstat -lntup | grep 9
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 95729/./../node/bin
tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 1471/slapd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 41931/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 45619/cupsd
tcp6 0 0 :::389 :::* LISTEN 1471/slapd
tcp6 0 0 :::22 :::* LISTEN 41931/sshd
tcp6 0 0 ::1:631 :::* LISTEN 45619/cupsd
tcp6 0 0 :::4728 :::* LISTEN 6259/caf
udp 0 0 0.0.0.0:47105 0.0.0.0:* 2094/sysedge
udp 0 0 0.0.0.0:1691 0.0.0.0:* 2094/sysedge
udp 0 0 0.0.0.0:28362 0.0.0.0:* 2094/sysedge
udp6 0 0 :::49753 :::* 2094/sysedge
udp6 0 0 :::1691 :::* 2094/sysedge
udp6 0 0 :::3002 :::* 6259/caf
I just notice in your setup, instead of doing tribe.t2.network.publish_host: ["172.16.153.14"], do this tribe.t2.network.publish_host: 172.16.153.14
ES might not "like" the ["host"] format for publish_host so it tried to do "localhost:port" instead of "specified-host:port"
My guess about "localhost:9400" is ES will try to bind to any port number in this range [9300 - 9400] if the one it needs to bind is in use.
In my setup , my tribe node uses port 9300 and 9200 (like your setup)
For t1 and t2, it uses other ports and in my setup, it uses 9301 and 9302 to communicate with two external clusters since 9300 is already in use by my tribe node's ES instance.
Tried tribe.t2.network.publish_host: 172.16.153.14. But having same error with 9400 port.
In your configuration, are you having t1, t2 and the tribe in same physical machine?
No, not at all.
Did you update both t1 and t2 or you only tried t2? To make it simple, comment out either t1 or t2 and try to make it work with one only. Once it's working, you can put the second cluster back on. Otherwise, you are chasing two different things.
Ultimately we could start the tribe with the following in elasticsearch.yml (lines with bold text are commented):
#network.host: 0.0.0.0
network.host: 172.16.153.14
transport.tcp.port: 9300
http.port: 9200
http.enabled: true
tribe.t1.cluster.name: pr_site
tribe.t1.node.name: pr_app2
tribe.t1.discovery.zen.ping.unicast.hosts: hydupiapp02:9300
tribe.t1.discovery.zen.ping.multicast.enabled: false
tribe.t1.path.conf: /home/npci/Desktop/UPI/elasticsearch-2.2.0/config
tribe.t1.path.plugins: /home/npci/Desktop/UPI/elasticsearch-2.2.0/plugins
#tribe.t1.network.bind_host: 0.0.0.0
#tribe.t1.network.publish_host: 172.16.153.14
But when I hit the Kibana, connecting to tribe, I get the follwoing in elasticsearch logs:
[2016-05-12 19:47:27,022][DEBUG][action.admin.indices.create] [Abdul Alhazred] no known master node, scheduling a retry
[2016-05-12 19:47:57,023][DEBUG][action.admin.indices.create] [Abdul Alhazred] timed out while retrying [indices:admin/create] after failure (timeout [30s])
[2016-05-12 19:47:57,025][INFO ][rest.suppressed ] /.kibana Params: {index=.kibana}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:205)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:794)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Kibana log has the following:
{"type":"log","@timestamp":"2016-05-12T14:23:28+00:00","tags":["status","plugin:elasticsearch","info"],"pid":114277,"name":"plugin:elasticsearch","state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Connecting kibana to the tribe node with this error is a different issue. I'm assuming your tribe node setup is working.
Search on this "elasticsearch" discussion, there is a thread that talks about hooking up kibana with tribe node.
Looks like adding multiple nodes from a tribe is not working for me. The tribe's elasticsearch.yml is as below:
node.name: tribe_node
network.host: 172.18.17.46
transport.tcp.port: 9300
http.port: 9200
http.enabled: true
tribe.t1.cluster.name: pr_site
tribe.t1.node.name: pr_node
tribe.t1.discovery.zen.ping.unicast.hosts: ["172.18.17.43:9300"]
tribe.t1.discovery.zen.ping.multicast.enabled: false
tribe.t1.path.conf: /home/npciadm/Desktop/NPCI/elasticsearch-2.1.1/config
tribe.t1.path.plugins: /home/npciadm/Desktop/NPCI/elasticsearch-2.1.1/plugins
tribe.t1.network.bind_host: 0.0.0.0
tribe.t1.network.publish_host: ["172.18.17.46"]
tribe.t2.cluster.name: ha_site
tribe.t2.node.name: ha_node
tribe.t2.discovery.zen.ping.unicast.hosts: ["172.18.17.44:9300"]
tribe.t2.discovery.zen.ping.multicast.enabled: false
tribe.t2.path.conf: /home/npciadm/Desktop/NPCI/elasticsearch-2.1.1/config
tribe.t2.path.plugins: /home/npciadm/Desktop/NPCI/elasticsearch-2.1.1/plugins
tribe.t2.network.bind_host: 0.0.0.0
tribe.t2.network.publish_host: ["172.18.17.46"]
But while starting the tribe, it is giving the following error. If I start the tribe using 9300, it tries to connect to 9301. If I start with 9301, it gives the error with 9300:
[2016-05-17 20:41:15,296][INFO ][node ] [tribe_node] version[2.1.1], pid[22695], build[40e2c53/2015-12-15T13:05:55Z]
[2016-05-17 20:41:15,296][INFO ][node ] [tribe_node] initializing ...
[2016-05-17 20:41:15,342][INFO ][plugins ] [tribe_node] loaded [], sites []
[2016-05-17 20:41:16,341][INFO ][node ] [tribe_node/t1] version[2.1.1], pid[22695], build[40e2c53/2015-12-15T13:05:55Z]
[2016-05-17 20:41:16,341][INFO ][node ] [tribe_node/t1] initializing ...
[2016-05-17 20:41:16,459][INFO ][plugins ] [tribe_node/t1] loaded [], sites []
[2016-05-17 20:41:16,543][INFO ][node ] [tribe_node/t1] initialized
[2016-05-17 20:41:16,544][INFO ][node ] [tribe_node/t2] version[2.1.1], pid[22695], build[40e2c53/2015-12-15T13:05:55Z]
[2016-05-17 20:41:16,544][INFO ][node ] [tribe_node/t2] initializing ...
[2016-05-17 20:41:16,569][INFO ][plugins ] [tribe_node/t2] loaded [], sites []
[2016-05-17 20:41:16,632][INFO ][node ] [tribe_node/t2] initialized
[2016-05-17 20:41:16,635][INFO ][node ] [tribe_node] initialized
[2016-05-17 20:41:16,635][INFO ][node ] [tribe_node] starting ...
[2016-05-17 20:41:16,677][INFO ][transport ] [tribe_node] publish_address {172.18.17.46:9300}, bound_addresses {172.18.17.46:9300}
[2016-05-17 20:41:16,681][INFO ][discovery ] [tribe_node] elasticsearch/hzedYqV3TvaWKOhuUEhMSg
[2016-05-17 20:41:16,681][WARN ][discovery ] [tribe_node] waited for 0s and no initial state was set by the discovery
[2016-05-17 20:41:16,687][INFO ][http ] [tribe_node] publish_address {172.18.17.46:9200}, bound_addresses {172.18.17.46:9200}
[2016-05-17 20:41:16,687][INFO ][node ] [tribe_node/t1] starting ...
[2016-05-17 20:41:16,695][INFO ][transport ] [tribe_node/t1] publish_address {127.0.0.1:9301}, bound_addresses {[::]:9301}
[2016-05-17 20:41:16,697][INFO ][discovery ] [tribe_node/t1] pr_site/JMA9-nQ6QEiPZ2v1diDXYg
[2016-05-17 20:41:19,737][INFO ][discovery.zen ] [tribe_node/t1] failed to send join request to master [{pr_node}{huT6N4kvR5mGXOsY0TG87w}{172.18.17.43}{172.18.17.43:9300}{master=true}], reason [RemoteTransportException[[pr_node][172.18.17.43:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[tribe_node/t1][127.0.0.1:9301] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[Connection refused: /127.0.0.1:9301]; ]
^C[2016-05-17 20:41:20,985][INFO ][node ] [tribe_node] stopping ...
[2016-05-17 20:41:20,998][INFO ][node ] [tribe_node] stopped
[2016-05-17 20:41:20,998][INFO ][node ] [tribe_node] closing ...
[2016-05-17 20:41:20,999][INFO ][node ] [tribe_node/t1] stopping ...
[2016-05-17 20:41:21,005][INFO ][node ] [tribe_node/t1] stopped
[2016-05-17 20:41:21,005][INFO ][node ] [tribe_node/t1] closing ...
[2016-05-17 20:41:21,009][INFO ][node ] [tribe_node/t1] closed
[2016-05-17 20:41:21,009][INFO ][node ] [tribe_node/t2] closing ...
[2016-05-17 20:41:21,012][INFO ][node ] [tribe_node/t2] closed
[2016-05-17 20:41:21,013][INFO ][node ] [tribe_node] closed
it should be tribe.t1.network.publish_host: 172.18.17.46
and the same goes to t2
Now I have made the tribe configuration as below:
node.name: tribe_node
network.host: 172.16.153.14
transport.tcp.port: 9300
http.port: 9200
http.enabled: true
tribe.t1.cluster.name: pr_site
tribe.t1.node.name: pr_app2
tribe.t1.discovery.zen.ping.unicast.hosts: ["172.16.173.12:9301"]
tribe.t1.discovery.zen.ping.multicast.enabled: false
tribe.t1.path.conf: /home/user1/Desktop/UPI/elasticsearch-2.2.0/config
tribe.t1.path.plugins: /home/user1/Desktop/UPI/elasticsearch-2.2.0/plugins
tribe.t1.network.bind_host: 0.0.0.0
tribe.t1.network.publish_host: "172.16.153.14"
tribe.t1.network.host: 0.0.0.0
tribe.t1.transport.tcp.port: 9301
Still getting the following error while starting the tribe. Can you please help:
[2016-06-09 10:54:27,904][INFO ][node ] [tribe_node] version[2.2.0], pid[44262], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-06-09 10:54:27,904][INFO ][node ] [tribe_node] initializing ...
[2016-06-09 10:54:28,236][INFO ][plugins ] [tribe_node] modules [lang-expression, lang-groovy], plugins [], sites []
[2016-06-09 10:54:29,203][INFO ][node ] [tribe_node/t1] version[2.2.0], pid[44262], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-06-09 10:54:29,203][INFO ][node ] [tribe_node/t1] initializing ...
[2016-06-09 10:54:29,370][INFO ][plugins ] [tribe_node/t1] modules [lang-expression, lang-groovy], plugins [], sites []
[2016-06-09 10:54:29,481][INFO ][node ] [tribe_node/t1] initialized
[2016-06-09 10:54:29,485][INFO ][node ] [tribe_node] initialized
[2016-06-09 10:54:29,485][INFO ][node ] [tribe_node] starting ...
[2016-06-09 10:54:29,544][INFO ][transport ] [tribe_node] publish_address {172.16.153.14:9300}, bound_addresses {172.16.153.14:9300}
[2016-06-09 10:54:29,547][INFO ][discovery ] [tribe_node] elasticsearch/GjpshninSm6cNthcAARsXw
[2016-06-09 10:54:29,547][WARN ][discovery ] [tribe_node] waited for 0s and no initial state was set by the discovery
[2016-06-09 10:54:29,567][INFO ][http ] [tribe_node] publish_address {172.16.153.14:9200}, bound_addresses {172.16.153.14:9200}
[2016-06-09 10:54:29,567][INFO ][node ] [tribe_node/t1] starting ...
[2016-06-09 10:54:29,608][INFO ][node ] [tribe_node/t1] stopping ...
[2016-06-09 10:54:29,609][INFO ][node ] [tribe_node/t1] stopped
[2016-06-09 10:54:29,609][INFO ][node ] [tribe_node/t1] closing ...
[2016-06-09 10:54:29,613][INFO ][node ] [tribe_node/t1] closed
Exception in thread "main" BindTransportException[Failed to bind to [9301]]; nested: ChannelException[Failed to bind to: /0.0.0.0:9301]; nested: AccessControlException[access denied ("java.net.SocketPermission" "localhost:9301" "listen,resolve")];
Likely root cause: java.security.AccessControlException: access denied ("java.net.SocketPermission" "localhost:9301" "listen,resolve")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkListen(SecurityManager.java:1131)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:221)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Refer to the log for complete error details.
[2016-06-09 10:54:29,617][INFO ][node ] [tribe_node] stopping ...
[2016-06-09 10:54:29,630][INFO ][node ] [tribe_node] stopped
[2016-06-09 10:54:29,631][INFO ][node ] [tribe_node] closing ...
[2016-06-09 10:54:29,632][INFO ][node ] [tribe_node] closed
if you read my previous post carefully, you should have this one fixed...
This configuration is working fine in LAN. Has anybody tried the tribe configuration in WAN. I am still getting the same error in WAN. All ports have been opened in firewall. Any clue will be helpful.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.