ES cluster with SSH and port forwarding, transport port 9300

Hello everyone!

I would like to set up an ES cluster consisting of three nodes, i.e. my local machine ('localhost') and two virtual machines in a remote network ('host1' and 'host2')

For each of them, I have this very simple elasticsearch.yml:

  network.host: 0.0.0.0
  discovery.zen.ping.unicast.hosts: ["localhost","host1","host2"]

The two remote nodes are already seeing each other:

curl -XGET 'localhost:9200/_cluster/health?pretty'
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 15,
  "active_shards" : 30,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Now, what I would like to do is to make my local machine to also see these nodes and be seen by them, hence having finally a 3 nodes cluster.

I assume that I am missing a parameter in my elasticsearch.yml file in order to do that.
Obviously, I will also have to connect my local machine to one of the nodes with an ssh and port forwarding, maybe like this:

ssh -L 9301:localhost:9300 root@host1

What am I missing? Thank you for your help!

Hello again! Is there really no one who can help me with this?

You might have forgotten to set network.host on your local machine?

No, it is set on all three machines.

Can you share the 3 elasticsearch.yml and the 3 node logs please?

elasticsearch.yml (same for all three):

 network.host: 0.0.0.0
 discovery.zen.ping.unicast.hosts: ["130.183.2.90","130.183.209.36","130.183.198.13","130.183.198.14"]
 transport.tcp.port: 9301

log local machine:

[2018-03-13T11:51:47,235][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-13T11:51:47,380][INFO ][o.e.e.NodeEnvironment    ] [k4P04zh] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [181.3gb], net total_space [450gb], types [ext4]
[2018-03-13T11:51:47,382][INFO ][o.e.e.NodeEnvironment    ] [k4P04zh] heap size [494.9mb], compressed ordinary object pointers [true]
[2018-03-13T11:51:47,417][INFO ][o.e.n.Node               ] node name [k4P04zh] derived from node ID [k4P04zhLSAaWtRw_vIuCXA]; set [node.name] to override
[2018-03-13T11:51:47,418][INFO ][o.e.n.Node               ] version[6.1.3], pid[11683], build[af51318/2018-01-26T18:22:55.523Z], OS[Linux/4.10.0-33-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-13T11:51:47,418][INFO ][o.e.n.Node               ] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [aggs-matrix-stats]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [analysis-common]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [ingest-common]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [lang-expression]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [lang-mustache]
[2018-03-13T11:51:48,029][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [lang-painless]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [mapper-extras]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [parent-join]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [percolator]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [reindex]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [repository-url]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [transport-netty4]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded module [tribe]
[2018-03-13T11:51:48,030][INFO ][o.e.p.PluginsService     ] [k4P04zh] loaded plugin [ingest-attachment]
[2018-03-13T11:51:49,027][INFO ][o.e.d.DiscoveryModule    ] [k4P04zh] using discovery type [zen]
[2018-03-13T11:51:49,502][INFO ][o.e.n.Node               ] initialized
[2018-03-13T11:51:49,502][INFO ][o.e.n.Node               ] [k4P04zh] starting ...
[2018-03-13T11:51:49,627][INFO ][o.e.t.TransportService   ] [k4P04zh] publish_address {130.183.2.90:9301}, bound_addresses {[::]:9301}
[2018-03-13T11:51:49,647][INFO ][o.e.b.BootstrapChecks    ] [k4P04zh] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-13T11:51:52,687][INFO ][o.e.c.s.MasterService    ] [k4P04zh] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {k4P04zh}{k4P04zhLSAaWtRw_vIuCXA}{uNT6gFa_Q4qJyJUC2SsToQ}{130.183.2.90}{130.183.2.90:9301}
[2018-03-13T11:51:52,692][INFO ][o.e.c.s.ClusterApplierService] [k4P04zh] new_master {k4P04zh}{k4P04zhLSAaWtRw_vIuCXA}{uNT6gFa_Q4qJyJUC2SsToQ}{130.183.2.90}{130.183.2.90:9301}, reason: apply cluster state (from master [master {k4P04zh}{k4P04zhLSAaWtRw_vIuCXA}{uNT6gFa_Q4qJyJUC2SsToQ}{130.183.2.90}{130.183.2.90:9301} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-13T11:51:52,725][INFO ][o.e.h.n.Netty4HttpServerTransport] [k4P04zh] publish_address {130.183.2.90:9200}, bound_addresses {[::]:9200}
[2018-03-13T11:51:52,726][INFO ][o.e.n.Node               ] [k4P04zh] started
[2018-03-13T11:51:53,140][INFO ][o.e.g.GatewayService     ] [k4P04zh] recovered [4] indices into cluster_state
[2018-03-13T11:51:55,554][INFO ][o.e.c.r.a.AllocationService] [k4P04zh] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).

log remote machine:

[2018-03-13T11:50:34,218][INFO ][o.e.n.Node               ] [] initializing ...
    [2018-03-13T11:50:34,326][INFO ][o.e.e.NodeEnvironment    ] [zp2mzxv] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [14.3gb], net total_space [19gb], types [rootfs]
    [2018-03-13T11:50:34,327][INFO ][o.e.e.NodeEnvironment    ] [zp2mzxv] heap size [193.3mb], compressed ordinary object pointers [true]
    [2018-03-13T11:50:34,360][INFO ][o.e.n.Node               ] node name [zp2mzxv] derived from node ID [zp2mzxv_QQGRk6QyzS5lCw]; set [node.name] to override
    [2018-03-13T11:50:34,360][INFO ][o.e.n.Node               ] version[6.2.2], pid[22558], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/3.10.0-693.5.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
    [2018-03-13T11:50:34,360][INFO ][o.e.n.Node               ] JVM arguments [-Xms200m, -Xmx200m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [aggs-matrix-stats]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [analysis-common]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [ingest-common]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [lang-expression]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [lang-mustache]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [lang-painless]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [mapper-extras]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [parent-join]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [percolator]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [rank-eval]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [reindex]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [repository-url]
    [2018-03-13T11:50:35,090][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [transport-netty4]
    [2018-03-13T11:50:35,091][INFO ][o.e.p.PluginsService     ] [zp2mzxv] loaded module [tribe]
    [2018-03-13T11:50:35,091][INFO ][o.e.p.PluginsService     ] [zp2mzxv] no plugins loaded
    [2018-03-13T11:50:37,980][INFO ][o.e.d.DiscoveryModule    ] [zp2mzxv] using discovery type [zen]
    [2018-03-13T11:50:38,574][INFO ][o.e.n.Node               ] initialized
    [2018-03-13T11:50:38,574][INFO ][o.e.n.Node               ] [zp2mzxv] starting ...
    [2018-03-13T11:50:38,722][INFO ][o.e.t.TransportService   ] [zp2mzxv] publish_address {130.183.198.13:9301}, bound_addresses {[::]:9301}
    [2018-03-13T11:50:38,740][INFO ][o.e.b.BootstrapChecks    ] [zp2mzxv] bound or publishing to a non-loopback address, enforcing bootstrap checks
    [2018-03-13T11:50:41,960][INFO ][o.e.c.s.ClusterApplierService] [zp2mzxv] detected_master {lx0UoSp}{lx0UoSpkS5eC2Pxoqw1beA}{DmWqzvu1R96A9GoQ0yapXw}{130.183.198.14}{130.183.198.14:9301}, added {{lx0UoSp}{lx0UoSpkS5eC2Pxoqw1beA}{DmWqzvu1R96A9GoQ0yapXw}{130.183.198.14}{130.183.198.14:9301},}, reason: apply cluster state (from master [master {lx0UoSp}{lx0UoSpkS5eC2Pxoqw1beA}{DmWqzvu1R96A9GoQ0yapXw}{130.183.198.14}{130.183.198.14:9301} committed version [10]])
    [2018-03-13T11:50:42,159][INFO ][o.e.h.n.Netty4HttpServerTransport] [zp2mzxv] publish_address {130.183.198.13:9200}, bound_addresses {[::]:9200}
    [2018-03-13T11:50:42,159][INFO ][o.e.n.Node               ] [zp2mzxv] started

Your port is 9301 but this:

discovery.zen.ping.unicast.hosts:["130.183.2.90","130.183.209.36","130.183.198.13","130.183.198.14"]

Means actually:

discovery.zen.ping.unicast.hosts:["130.183.2.90:9300","130.183.209.36:9300","130.183.198.13:9300","130.183.198.14:9300"]

So change with:

discovery.zen.ping.unicast.hosts:["130.183.2.90:9301","130.183.209.36:9301","130.183.198.13:9301","130.183.198.14:9301"]

I see. And what kind of port forwarding should I do in that case?

(Just refreshing the topic :slight_smile: )

And what kind of port forwarding should I do in that case?

Sorry but I don't know. May be someone else.

Hi @Maxclac

Why do you want to forward those ports ?

Access them from outside of your local network ? (WAN)

Yes. :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.