# Overriding \`tcp.publish\_port\` breaks clustering when elasticsearch is in a container
**URL:** https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135
**Category:** Elasticsearch
**Created:** [June 8, 2016, 12:05am UTC](https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135 "2016-06-08T00:05:03Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@xavi](https://discuss.elastic.co/u/xavi)
#### Post date: [June 8, 2016, 12:05am UTC](https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135/1 "2016-06-08T00:05:04Z")
I'm trying to run an elasticsearch cluster with each es-node running in its own container. These containers are deployed using [ECS](https://aws.amazon.com/ecs/) across several machines that may be running other unrelated containers. To avoid port conflicts each port a container exposes is assigned a random value. These random ports are consistent across all running containers of the same type. In other words, all running es-node containers map port 9300 to the same random number.
Here's the config I'm using:
```
network:
host: 0.0.0.0
plugin:
mandatory: cloud-aws
cluster:
name: ${ES_CLUSTER_NAME}
discovery:
type: ec2
ec2:
groups: ${ES_SECURITY_GROUP}
any_group: false
zen.ping.multicast.enabled: false
transport:
tcp.port: 9300
publish_port: ${_INSTANCE_PORT_TRANSPORT}
cloud.aws:
access_key: ${AWS_ACCESS_KEY}
secret_key: ${AWS_SECRET_KEY}
region: ${AWS_REGION}
```
In this case `_INSTANCE_PORT_TRANSPORT` is the port that 9300 is bound to on the host machine. I've confirmed that all the environment variables used above are set correctly. I'm also setting `network.publish_host` to the host machine's local IP via a command line arg.
When I forced `_INSTANCE_PORT_TRANSPORT` (and in turn `transport.publish_port`) to be 9300, everything worked great, but as soon as it's given a random value, nodes can no longer connect to each other. I see errors like this using `logger.discovery=TRACE`:
```
ConnectTransportException[[][10.0.xxx.xxx:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.xxx.xxx:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
It seems like the port a node binds to is the same as the port it pings while trying to connect to other nodes. Is there any way to make them different? If not, what's the point of `transport.publish_port`?
Here are the full logs: [https://gist.github.com/xavi-/6ecc4ba16b39680fb28c8fb25307bcc7](https://gist.github.com/xavi-/6ecc4ba16b39680fb28c8fb25307bcc7)
---
### Author:  [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)
#### Post date: [June 15, 2016, 2:20pm UTC](https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135/2 "2016-06-15T14:20:24Z")
Hi @xavi,
I've seen that you've [asked the same question on Stackoverfllow](http://stackoverflow.com/questions/37668141/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-conta/37706726) and I'm adding it just for reference.
Neither ES nor the AWS plugin are container-aware. So if you map the container port 9300 to the external port 9300 this just worked because both used the same port. If the mapping is different the typical solution is to use a dedicated cluster manager that "knows" the cluster topology. These [blog article about Docker networking](https://www.elastic.co/blog/docker-networking) from our engineering team (and the [Docker overlay networking](https://docs.docker.com/engine/userguide/networking/dockernetworks/#an-overlay-network) mentioned in the article) should get you started.
Another solution could be the [Elasticsearch Mesos framework](https://github.com/mesos/elasticsearch) (_not_ maintained by Elastic).
Daniel
---
### Author:  [@xavi](https://discuss.elastic.co/u/xavi)
#### Post date: [September 28, 2016, 8:17pm UTC](https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135/3 "2016-09-28T20:17:09Z")
I don't think you understand the problem. In a clusters, there's a no guarantee that port 9300 (or any specific port) is available on a host machine at deploy time. Yes, one possible solution is to give each container an IP, but unfortunately that's not supported in amazon ECS. Instead the solution amazon uses is to map each port a container requests to a random host port.
So, for example, container port 9300 may get mapped to host port 10137. As a result you effectively end up with a ES process that can only receive TCP connections on port 10137, but boardcasts that other nodes should use port 9300 when connecting to it. This obviously broken. It doesn't sound like ES has a viable solution, which is regrettable.
---
### Author:  [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)
#### Post date: [October 19, 2016, 10:41am UTC](https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135/4 "2016-10-19T10:41:15Z")
Hi,
> [@xavi](#):
>
> I don't think you understand the problem.
I got the problem but my point still holds: None of the plugins is container-aware and for a good reason. The standard practice is to use a dedicated cluster manager. The container should be treated as a jail and the software running inside the container shouldn't be aware of the host that it is running on (among others for security reasons).
> [@xavi](#):
>
> the solution amazon uses is to map each port a container requests to a random host port.
This is true but you can also specify a port mapping, when you create a task (see [ECS docs](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html)). So if you don't want to use a cluster manager, I'd suggest you go down that route.
I hope that helps.
Daniel
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:11pm UTC](https://discuss.elastic.co/t/overriding-tcp-publish-port-breaks-clustering-when-elasticsearch-is-in-a-container/52135/5 "2017-07-05T22:11:19Z")