Upgrading to ES 2.2.0 doesn't allow node to have different bind and publish addresses

Elastic Stack Elasticsearch
1

Just tried to upgrade from 2.1.0 to 2.2.0 but having a problem with the network bind_host and publish_host settings.

In 2.1.0, we have this config. Bind_host is set to loopback so that it cannot be connected to remotely (over http) as we run the node behind a reverse proxy, and publish_host set to external interface.

network.bind_host: 127.0.0.1
network.publish_host: myhostname
transport.bind_host: myhostname

This worked just fine, but when I try this in 2.2.0, the node starts and then dies with a message like this : (myhostname and 1.2.3.4 hashing out real values)

[2016-02-09 15:14:09,114][ERROR][bootstrap                ] [myhostname] Exception
BindHttpException[Publish address [myhostname/1.2.3.4] does not match any of the bound addresses [[127.0.0.1:9200]]]

It seems that bind and publish values must now be the same, which kind of breaks things as I can't secure the listener to loopback without also setting the publish host to loopback which means the node can't talk to anything else.

Any way round this?

2

Try setting an actual IP or interface.

3

If you study the change

you can probably work around that by assigning a fixed HTTP port number.

4

Still no joy.

I've tried various settings including

network.publish_host: 1.2.3.4
http.port: 9200
transport.tcp.port: 9300

Same error each time :

Exception in thread "main" BindHttpException[Publish address [/1.2.3.4] does not match any of the bound addresses [[127.0.0.1:9200]]]
	at org.elasticsearch.http.netty.NettyHttpServerTransport.doStart(NettyHttpServerTransport.java:279)

I looked at the code but confess to not being a developer and so don't really understand what it's doing under the hood to know what else to change.

5

For me, this is a major issue. I prevents me from using elasticsearch 2.2, because my docker setup relies on the bind host being different from the publish host. Published host is the container host's IP so the rest of the world can find the node, but it's not allowed to bind to it.

6

Yes, I agree, to me it's also a major issue.

Seems I can not deploy on Docker anymore :frowning:

I can also see the bug in this example

  "network" : {
     "host" : "1.2.3.4",
     "port" : 9302
  },
  "http" : {
     "bind_host" : "0.0.0.0",
     "publish_host" : "1.2.3.4",
     "port" : 9203
  },

ES tries to bind to 0.0.0.0:9200 (!) which is definitely ignoring http.port.

1 Like
7

The setting which needs to be set is http.publish_port.

http.port is used for the bound address and http.publish_port for the publish address.

The error message is unfortunately misleading in this case. The reason this check was put in place is that previously, when there were multiple bound addresses (with possibly different ports) and a publish address not matching any of the bound addresses, we would just randomly pick one. Instead of making the fallback logic smarter (which is already pretty complex), we opted to fail fast. With that said, I'll have another look at the code and see how we can better support the Docker scenario.

2 Likes
8

First and foremost. This change broke the cardinal rule of "If it ain't broke don't fix it!"

I've lost nearly 3 hours this morning due to this poorly executed and equally poorly publicizedhttps://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-2.2.html code revision due to a minor version update, which happens automatically during an apt-get update && apt-get upgrade!

I finally got it working by specifying the "site" addresses in the network.host setting:

network.host: ["_local_","_site_"]
network.publish_host: "<redacted>" # used for all node-to-node communication

These are the kinds of things that continuously prevent me from upgrading my critical production environments beyond 1.7.x.

1 Like
9

Did you jump directly from 1.7 to 2.2? Did you reach the breaking changes for the initial 2.0 release?

10

No. My production environment is on 1.7.5.

This is my development build. It was running 2.1

11

Just a quick update, I got this working by using the following parameters in my elasticsearch.yml file

network.host: "myhostname"
http.bind_host: 127.0.0.1
http.publish_port: 9200
http.port: 9200

This limits incoming HTTP requests to local host, but allows the node to discover and connect to other nodes in the cluster. Not a totally secure solution, I know, but it at least brings us back to where we were and allows us to update to 2.2.0

1 Like
12

@schmorgs just adding http.publish_port to your initial config should do the trick, e.g.

network.bind_host: 127.0.0.1
network.publish_host: myhostname
transport.bind_host: myhostname
http.publish_port: 9200
1 Like
13

I had the same problem and i resolved it with your Config. Thanks
The problem was that ES gives automatically an other bind_host adresse, because i didn't configure it
network.host: "192.168.1.40"
http.bind_host: 127.0.0.1
http.publish_port: 9200
http.port: 9200
now i got the two nodes in the cluster (y)