Problem with join additional nodes to dockerized cluster elasticsearch 6.6.2 + xpack

[amilek]

Hello, everybody!

I try to deploy this configuration:
ES 6.6.2 on several docker hosts, 3 nodes.
Two nodes are master-eligible and one node is "data only".
XPACK enabled (basic license).
Inter node communication is encrypted by "p12" certificate with "xpack.security.transport.ssl.verification_mode=certificate". Actually, I generated one common "p12" certificate for all nodes

docker host1 .env: https://pastebin.com/CEXSMJD0

docker host2 .env: https://pastebin.com/wWvcJ7xn

docker host3 .env: https://pastebin.com/NYKcLSTp

docker-compose.yml: https://pastebin.com/yuGkcmDe

When I run host1's docker-compose and host2's docker-compose - cluster "es2" up an running "green", with "elastic2" as "master" node.

Next step I trying to add additional node "elastic3" as "data only" node.
On master node "elastic2" I can see in logs: https://pastebin.com/mFm1vuDx
On other master eligible node "elastic1" I can see in logs: https://pastebin.com/YG87rmyd

The problem also occured on dockerized ES version 6.7.2. (done it just for test)

If I change "DISCOVERY_ZEN_MINIMUM_MASTER_NODES" to "3" and "NODE_MASTER" to "true" (on host3's .env file)- cluster up and running "green" with 3 nodes. (I don't like this choice because if only one node fall down - cluster cannot elect "master node")

If I will change option "xpack.security.enabled" to "false" in all docker-compose.yml files across all nodes - cluster up and running "green" with 3 nodes. (I don't like this choice because I need some security options from xpack)

If I upgrade version of ES to 6.8.3 (just change version in all docker-compose.yml files) - cluster up and running "green" with 3 nodes. (I cannot use version upper than 6.6.2 because of some breaking changes in perl ES module.)

I am running out of ideas, please help.

With best regards,
Amil.

[amilek]

I will ask to myself:

prior to version 6.8.0 the "security" features of "xpack" was NOT free with "basic" license.
Enabling security features of "xpack" in 6.6.2 version leads to such strange errors:
zen-disco-node-failed on "master" node;
o.e.c.s.ClusterApplierService removed on "master" node;
and
general node connection failure on "master eligible" node.

proof: https://www.elastic.co/blog/security-for-elasticsearch-is-now-free