Transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]

[dsosunov (Dmitriy Sosunov)]

We recently deploy a new cluster v7.4 and receiving very strange deprecation warnings.

[2019-10-07T07:08:33,504][WARN ][o.e.d.t.TransportInfo ] [node-01.example.com] transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.
[2019-10-07T07:08:33,504][WARN ][o.e.d.h.HttpInfo ] [node-01.example.com] es.http.cname_in_publish_address system property is deprecated and no longer affects http.publish_address formatting. Remove this property to get rid of this deprecation warning.

I started to review which address was actually published:
curl "https://node-02.example.com.com:9200/_nodes" | jq ".nodes.http.publish_address"

"node-07.example.com/10.13.0.160:9200"
"node-06.example.com/10.13.0.151:9200"
"node-08.example.com/10.13.0.143:9200"
"node-01.example.com/10.13.0.51:9200"
"node-05.example.com/10.13.0.141:9200"
"node-03.example.com/10.13.0.53:9200"
"node-04.example.com/10.13.0.55:9200"
"node-02.example.com/10.13.0.52:9200"

As for me, looks good.

The second one was:
/etc/elasticsearch/jvm.options

There are no extra options such as mentioned in warning.

The actual command line is:
/usr/share/elasticsearch/jdk/bin/java -Xms256m -Xmx256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=134217728 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

Please help me get rid of this warnings.

It's important for us, because we are going to migrate to ESv7 and there are a lot of systems that are using deprecated features, this warning produce a lot of noise.

[DavidTurner]

This is issue #47436 and will be addressed in 7.4.1.

[dsosunov (Dmitriy Sosunov)]

Thanks, for quick reply

But we have two warnings.
The issue #47436 it's about es.http.cname_in_publish_address system property is deprecated

The second one it's warning about publish address in transport section.

curl "https://node-02.example.com:9200/_nodes" | jq ".nodes.transport.publish_address"

"10.13.0.143:9300"
"10.13.0.51:9300"
"10.13.0.141:9300"
"10.13.0.160:9300"
"10.13.0.55:9300"
"10.13.0.151:9300"
"10.13.0.52:9300"
"10.13.0.53:9300"

and there only IP addresses instead of expected host/ip

I've double check configuration and there only two options that some how can affect transport address it's:

network.host: node-02.example.com

in general all options related to other hosts in the cluster are defined by full name (no CNAME);

Is this warning has the same roots or it's can be somehow fixed by other way?

[DavidTurner]

I see, sorry, your message is not well formatted so it's difficult to read both log messages. The one about the transport publish address is correct and instructs you how to resolve it:

@dsosunov

Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.

[dsosunov (Dmitriy Sosunov)]

@DavidTurner I tried to add this option.

The command line become:
/usr/share/elasticsearch/jdk/bin/java -Xms256m -Xmx256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.transport.cname_in_publish_address=true -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=134217728 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

and in logs I see:
[2019-10-07T07:55:24,000][WARN ][o.e.d.h.HttpInfo ] [node-01] es.http.cname_in_publish_address system property is deprecated and no longer affects http.publish_address formatting. Remove this property to get rid of this deprecation warning.

Any workarounds?
We are considering to filter such kind of noise in filebeat processors

processors:

• drop_event:
  when:
  regexp:
  message: ".es.http.cname_in_publish_address."

[DavidTurner]

I don't have the code in front of me right now, but if I remember correctly you can set es.http.cname_in_publish_address to false to suppress this warning until 7.4.1 comes out.