Transport publishing names vs new +7.0 format and es.transport.cname_in_publish_address

stefws · January 9, 2020, 8:04am

Got a 7.5.1 cluster of 12 nodes where the elected master node logs deprecation warnings like these per cluster node very frequently:

[2020-01-09T08:52:41,809][WARN ][o.e.d.t.TransportInfo ] [es-mst2] transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.

Using -Des.transport.cname_in_publish_address=true only makes warning log rate twice as it then also complains of deprecated use of just that option

Got these yaml settings:

network.host: ${HOSTNAME}, _local_
network.publish_host: ${HOSTNAME}

it logs on launch as expected =>

[2020-01-09T08:36:40,599][INFO ][o.e.t.TransportService   ] [es-mst2] publish_address {es-mst2/<ip>:9300}, bound_addresses {127.0.0.1:9300}, {<ip>:9300}
[2020-01-09T08:36:47,747][INFO ][o.e.h.AbstractHttpServerTransport] [es-mst2] publish_address {es-mst2/<ip>:9200}, bound_addresses {127.0.0.1:9200}, {<ip>:9200}

But still only find IPs published for transport:

# curl -s -XGET 'http://localhost:9200/_nodes' | jq '.nodes[].transport.publish_address'
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"

Http looks fine:

# curl -s -XGET  'http://localhost:9200/_nodes' | jq '.nodes[].http.publish_address'
"d1r2n12/<ip>:9200"
"es-mst3/<ip>:9200"
"es-mst2/<ip>:9200"
"d1r1n14/<ip>:9200"
"d1r1n11/<ip>:9200"
"es-i2/<ip>:9200"
"es-i3/<ip>:9200"
"es-mst1/<ip>:9200"
"d1r2n14/<ip>:9200"
"es-i1/<ip>:9200"
"d1r2n11/<ip>:9200"
"d1r1n12/<ip>:9200"

Appreciates any hints on how to stop these deprecation warnings, TIA!

stefws · January 9, 2020, 8:36am

Weirdly asking one of the data nodes it replies properly:

# curl -s -XGET 'http://d1r1n11:9200/_nodes' | jq '.nodes[].transport.publish_address'
"es-i2/<ip>:9300"
"es-mst3/<ip>:9300"
"d1r2n12/<ip>:9300"
"es-i1/<ip>:9300"
"es-mst2/<ip>:9300"
"d1r1n11/<ip>:9300"
"d1r2n14/<ip>:9300"
"d1r1n12/<ip>:9300"
"d1r1n14/<ip>:9300"
"es-i3/<ip>:9300"
"d1r2n11/<ip>:9300"
"es-mst1/<ip>:9300"

But all master+pure-ingest nodesreplies only with IPs...

DavidTurner · January 9, 2020, 9:08am

Please could you share the exact log messages you're seeing? There are a number of similar-looking messages and it's not clear exactly which ones you're asking about.

Could you also share the first few log messages from when your node started up? I am particularly interested with all of the messages before the sequence of loaded module ones.

stefws · January 9, 2020, 9:15am

Yeap warnings like this from deprecation log:

[2020-01-09T10:10:36,192][WARN ][o.e.d.t.TransportInfo    ] [es-mst2] transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.

Launch info log pre loaded module:

[2020-01-09T09:22:43,325][INFO ][o.e.e.NodeEnvironment    ] [es-mst2] using [1] data paths, mounts [[/var (/dev/mapper/vg_vm-lv_var)]], net usable_space [2.5gb], net total_space [3gb], types [ext4]
[2020-01-09T09:22:43,328][INFO ][o.e.e.NodeEnvironment    ] [es-mst2] heap size [3gb], compressed ordinary object pointers [true]
[2020-01-09T09:22:43,774][INFO ][o.e.n.Node               ] [es-mst2] node name [es-mst2], node ID [Wst-8O6sTFmh4umBIEI8SA], cluster name [mxes2data]
[2020-01-09T09:22:43,775][INFO ][o.e.n.Node               ] [es-mst2] version[7.5.1], pid[15989], build[default/rpm/3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96/2019-12-16T22:57:37.835892Z], OS[Linux/4.18.3-1.el6.elrepo.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2020-01-09T09:22:43,775][INFO ][o.e.n.Node               ] [es-mst2] JVM home [/usr/share/elasticsearch/jdk]
[2020-01-09T09:22:43,775][INFO ][o.e.n.Node               ] [es-mst2] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms3g, -Xmx3g, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-913920297370723851, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=3,filesize=10m, -Djava.locale.providers=COMPAT, -XX:UseAVX=0, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]

DavidTurner · January 9, 2020, 9:38am

The JVM arguments line does not include -Des.transport.cname_in_publish_address=true? If you set that option, what happens?

stefws · January 9, 2020, 9:44am

Then it complains that this option no longer works on http publishing

   {"type": "deprecation", "timestamp": "2020-01-09T08:28:51,051+01:00", "level": "WARN", "component": "o.e.d.h.HttpInfo", "cluster.name": "mxes2data", "node.name": "es-mst2", "message": "es.http.cname_in_publish_address system property is deprecated and no longer affects http.publish_address formatting. Remove this property to get rid of this deprecation warning.", "cluster.uuid": "SmpMKhEwQs2r0G4eNOBJTA", "node.id": "Wst-8O6sTFmh4umBIEI8SA"  }
   {"type": "deprecation", "timestamp": "2020-01-09T08:28:51,052+01:00", "level": "WARN", "component": "o.e.d.t.TransportInfo", "cluster.name": "mxes2data", "node.name": "es-mst2", "message": "transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.", "cluster.uuid": "SmpMKhEwQs2r0G4eNOBJTA", "node.id": "Wst-8O6sTFmh4umBIEI8SA"  }

DavidTurner · January 9, 2020, 9:45am

These messages are about different properties. The first message is about es.http.cname_in_publish_address which is deprecated and should be removed, but the second one is about es.transport.cname_in_publish_address which is not deprecated and should be added.

stefws · January 9, 2020, 11:05am

Right need to do proper spell checking, thanks...

Only I'm still wondering why data+ingest+ml nodes reports nodes as their names as claimed for transport but not master-only/ingest-only nodes?

[2020-01-09T08:36:40,599][INFO ][o.e.t.TransportService ] [es-mst2] publish_address {es-mst2/:9300}, bound_addresses {127.0.0.1:9300}, {:9300}

querying a data node I see:

# curl -s -XGET 'http://d1r1n11:9200/_nodes' | jq '.nodes[].transport.publish_address'
"es-i2/<ip>:9300"
"es-mst3/<ip>:9300"
"d1r2n12/<ip>:9300"
"es-i1/<ip>:9300"
"es-mst2/<ip>:9300"
"d1r1n11/<ip>:9300"
"d1r2n14/<ip>:9300"
"d1r1n12/<ip>:9300"
"d1r1n14/<ip>:9300"
"es-i3/<ip>:9300"
"d1r2n11/<ip>:9300"
"es-mst1/<ip>:9300"

where as local master node 2 says:

# curl -s -XGET 'http://localhost:9200/_nodes' | jq '.nodes[].transport.publish_address'
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"
"<ip>:9300"

DavidTurner · January 9, 2020, 12:02pm

Is -Des.transport.cname_in_publish_address=true on the data node d1r1n11? Check something like this:

curl -s 'http://localhost:9200/_nodes/d1r1n11/jvm' | jq .nodes[].jvm.input_arguments | grep cname_in_publish_address

stefws · January 9, 2020, 12:08pm

Yes they were added in jvm.options on all data nodes sometime ago it appears.

Entering same option on all nodes of the cluster makes them report in new +7 format also. Would have thought this now was default in +7.5 maybe, anyway thanks assisting in my _search

DavidTurner · January 9, 2020, 12:27pm

No problem, it was just as confusing to me when I first hit this.

The new format is available from 7.4.0 onwards, in preparation for changing the default in 8.0. We cannot change to the new format by default in any 7.x version since that would break any clients that rely on the pre-7.4 format, and we avoid making breaking changes in minor versions.

system · February 6, 2020, 12:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.