ES cluster with SSH and port forwarding, transport port 9300

Maxclac · February 27, 2018, 11:38am

Hello everyone!

I would like to set up an ES cluster consisting of three nodes, i.e. my local machine ('localhost') and two virtual machines in a remote network ('host1' and 'host2')

For each of them, I have this very simple elasticsearch.yml:

  network.host: 0.0.0.0
  discovery.zen.ping.unicast.hosts: ["localhost","host1","host2"]

The two remote nodes are already seeing each other:

curl -XGET 'localhost:9200/_cluster/health?pretty'
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 15,
  "active_shards" : 30,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Now, what I would like to do is to make my local machine to also see these nodes and be seen by them, hence having finally a 3 nodes cluster.

I assume that I am missing a parameter in my elasticsearch.yml file in order to do that.
Obviously, I will also have to connect my local machine to one of the nodes with an ssh and port forwarding, maybe like this:

ssh -L 9301:localhost:9300 root@host1

What am I missing? Thank you for your help!

Maxclac · March 12, 2018, 4:50pm

Hello again! Is there really no one who can help me with this?

dadoonet · March 12, 2018, 7:38pm

You might have forgotten to set network.host on your local machine?

Maxclac · March 13, 2018, 8:47am

No, it is set on all three machines.

dadoonet · March 13, 2018, 10:35am

Can you share the 3 elasticsearch.yml and the 3 node logs please?

Maxclac · March 13, 2018, 11:04am

elasticsearch.yml (same for all three):

 network.host: 0.0.0.0
 discovery.zen.ping.unicast.hosts: ["130.183.2.90","130.183.209.36","130.183.198.13","130.183.198.14"]
 transport.tcp.port: 9301

Maxclac · March 13, 2018, 11:09am

log local machine:

[2018-03-13T11:51:47,235][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-13T11:51:47,380][INFO ][o.e.e.NodeEnvironment    ] [k4P04zh] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [181.3gb], net total_space [450gb], types [ext4]
[2018-03-13T11:51:47,382][INFO ][o.e.e.NodeEnvironment    ] [k4P04zh] heap size [494.9mb], compressed ordinary object pointers [true]
[2018-03-13T11:51:47,417][INFO ][o.e.n.Node               ] node name [k4P04zh] derived from node ID [k4P04zhLSAaWtRw_vIuCXA]; set [node.name] to override
[2018-03-13T11:51:47,418][INFO ][o.e.n.Node               ] version[6.1.3], pid[11683], build[af51318/2018-01-26T18:22:55.523Z], OS[Linux/4.10.0-33-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-13T11:51:47,418][INFO ][o.e.n.Node               ] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [aggs-matrix-stats]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [analysis-common]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [ingest-common]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [lang-expression]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [lang-mustache]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [lang-painless]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [mapper-extras]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [parent-join]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [percolator]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [reindex]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [repository-url]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [transport-netty4]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [tribe]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded plugin [ingest-attachment]
[2018-03-13T11:51:49,027][INFO ][o.e.d.DiscoveryModule    ] [k4P04zh] using discovery type [zen]
[2018-03-13T11:51:49,502][INFO ][o.e.n.Node               ] initialized
[2018-03-13T11:51:49,502][INFO ][o.e.n.Node               ] [k4P04zh] starting ...
[2018-03-13T11:51:49,627][INFO ][o.e.t.TransportService   ] [k4P04zh] publish_address {130.183.2.90:9301}, bound_addresses {[::]:9301}
[2018-03-13T11:51:49,647][INFO ][o.e.b.BootstrapChecks    ] [k4P04zh] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-13T11:51:52,687][INFO ][o.e.c.s.MasterService    ] [k4P04zh] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {k4P04zh}{k4P04zhLSAaWtRw_vIuCXA}{uNT6gFa_Q4qJyJUC2SsToQ}{130.183.2.90}{130.183.2.90:9301}
[2018-03-13T11:51:52,692][INFO ][o.e.c.s.ClusterApplierService] [k4P04zh] new_master {k4P04zh}{k4P04zhLSAaWtRw_vIuCXA}{uNT6gFa_Q4qJyJUC2SsToQ}{130.183.2.90}{130.183.2.90:9301}, reason: apply cluster state (from master [master {k4P04zh}{k4P04zhLSAaWtRw_vIuCXA}{uNT6gFa_Q4qJyJUC2SsToQ}{130.183.2.90}{130.183.2.90:9301} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-13T11:51:52,725][INFO ][o.e.h.n.Netty4HttpServerTransport] [k4P04zh] publish_address {130.183.2.90:9200}, bound_addresses {[::]:9200}
[2018-03-13T11:51:52,726][INFO ][o.e.n.Node               ] [k4P04zh] started
[2018-03-13T11:51:53,140][INFO ][o.e.g.GatewayService     ] [k4P04zh] recovered [4] indices into cluster_state
[2018-03-13T11:51:55,554][INFO ][o.e.c.r.a.AllocationService] [k4P04zh] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).

Maxclac · March 13, 2018, 11:10am

log remote machine:

[2018-03-13T11:50:34,218][INFO ][o.e.n.Node               ] [] initializing ...
    [2018-03-13T11:50:34,326][INFO ][o.e.e.NodeEnvironment    ] [zp2mzxv] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [14.3gb], net total_space [19gb], types [rootfs]
    [2018-03-13T11:50:34,327][INFO ][o.e.e.NodeEnvironment    ] [zp2mzxv] heap size [193.3mb], compressed ordinary object pointers [true]
    [2018-03-13T11:50:34,360][INFO ][o.e.n.Node               ] node name [zp2mzxv] derived from node ID [zp2mzxv_QQGRk6QyzS5lCw]; set [node.name] to override
    [2018-03-13T11:50:34,360][INFO ][o.e.n.Node               ] version[6.2.2], pid[22558], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/3.10.0-693.5.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
    [2018-03-13T11:50:34,360][INFO ][o.e.n.Node               ] JVM arguments [-Xms200m, -Xmx200m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [aggs-matrix-stats]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [analysis-common]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [ingest-common]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [lang-expression]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [lang-mustache]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [lang-painless]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [mapper-extras]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [parent-join]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [percolator]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [rank-eval]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [reindex]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [repository-url]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [transport-netty4]
    [2018-03-13T11:50:35,091][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [tribe]
    [2018-03-13T11:50:35,091][INFO ][o.e.p.PluginsService     ] [zp2mzxv] no plugins loaded
    [2018-03-13T11:50:37,980][INFO ][o.e.d.DiscoveryModule    ] [zp2mzxv] using discovery type [zen]
    [2018-03-13T11:50:38,574][INFO ][o.e.n.Node               ] initialized
    [2018-03-13T11:50:38,574][INFO ][o.e.n.Node               ] [zp2mzxv] starting ...
    [2018-03-13T11:50:38,722][INFO ][o.e.t.TransportService   ] [zp2mzxv] publish_address {130.183.198.13:9301}, bound_addresses {[::]:9301}
    [2018-03-13T11:50:38,740][INFO ][o.e.b.BootstrapChecks    ] [zp2mzxv] bound or publishing to a non-loopback address, enforcing bootstrap checks
    [2018-03-13T11:50:41,960][INFO ][o.e.c.s.ClusterApplierService] [zp2mzxv] detected_master {lx0UoSp}{lx0UoSpkS5eC2Pxoqw1beA}{DmWqzvu1R96A9GoQ0yapXw}{130.183.198.14}{130.183.198.14:9301}, added {{lx0UoSp}{lx0UoSpkS5eC2Pxoqw1beA}{DmWqzvu1R96A9GoQ0yapXw}{130.183.198.14}{130.183.198.14:9301},}, reason: apply cluster state (from master [master {lx0UoSp}{lx0UoSpkS5eC2Pxoqw1beA}{DmWqzvu1R96A9GoQ0yapXw}{130.183.198.14}{130.183.198.14:9301} committed version [10]])
    [2018-03-13T11:50:42,159][INFO ][o.e.h.n.Netty4HttpServerTransport] [zp2mzxv] publish_address {130.183.198.13:9200}, bound_addresses {[::]:9200}
    [2018-03-13T11:50:42,159][INFO ][o.e.n.Node               ] [zp2mzxv] started

dadoonet · March 13, 2018, 12:17pm

Your port is 9301 but this:

discovery.zen.ping.unicast.hosts:["130.183.2.90","130.183.209.36","130.183.198.13","130.183.198.14"]

Means actually:

discovery.zen.ping.unicast.hosts:["130.183.2.90:9300","130.183.209.36:9300","130.183.198.13:9300","130.183.198.14:9300"]

So change with:

discovery.zen.ping.unicast.hosts:["130.183.2.90:9301","130.183.209.36:9301","130.183.198.13:9301","130.183.198.14:9301"]

Maxclac · March 13, 2018, 12:53pm

I see. And what kind of port forwarding should I do in that case?

Maxclac · March 20, 2018, 3:46pm

(Just refreshing the topic )

dadoonet · March 20, 2018, 6:12pm

And what kind of port forwarding should I do in that case?

Sorry but I don't know. May be someone else.

Ben96 · March 21, 2018, 9:43am

Hi @Maxclac

Why do you want to forward those ports ?

Access them from outside of your local network ? (WAN)

Maxclac · March 21, 2018, 7:53pm

Yes.

system · April 18, 2018, 7:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Answer for: ES cluster with SSH and port forwarding, transport port 9300 Elasticsearch	1	838	October 5, 2018
Configure cluster Elasticsearch	13	9442	April 6, 2017
Setting up 2-Node cluster on local network Elasticsearch	2	253	September 3, 2021
Adding a second node to cluster Elasticsearch	11	2615	May 24, 2018
Setting up elasticsearch.yml for Discovery Elasticsearch	7	2396	July 6, 2017

ES cluster with SSH and port forwarding, transport port 9300

Related topics