Hi there,
Im having an issue trying to add a new server to an existing cluster thats is using encription and Im getting the following error.
[2019-09-03T20:11:10,344][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elastic-kibana]
master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster,
and [cluster.initial_master_nodes] is empty on this node: have discovered [{elastic01}
{Q77grcs6Q2uIbcrGllKcyQ}{YXvwWhRjT6OXOq1zhe8cBQ}{10.2.208.26}{10.2.208.26:9300}
{ml.machine_memory=33742123008, ml.max_open_jobs=20, xpack.installed=true}, {elastic02}
{FT0SSbtQQkOvIoh7qwzvYg}{ooFyk1dqQ36hNK7CojUABw}{10.2.208.27}{10.2.208.27:9300}
{ml.machine_memory=33742123008, ml.max_open_jobs=20, xpack.installed=true}, {elastic03}
{8vIBMx9ZTRq2USZ4n295ag}{sWsPggPHQcO_dE3P6zGXUA}{10.2.208.28}{10.2.208.28:9300}
{ml.machine_memory=33742123008, ml.max_open_jobs=20, xpack.installed=true}]; discovery
will continue using [10.2.208.26:9300, 10.2.208.27:9300, 10.2.208.28:9300] from hosts providers
and [{elastic-kibana}{sWb4AwzaRPGWmxJYCo3Sgw}{svrZlwvJSeC2J2P6RYmRwQ}{10.4.28.35}
{10.4.28.35:9300}{ml.machine_memory=8375558144, xpack.installed=true,
ml.max_open_jobs=20}] from last-known cluster state; node term 18, last-accepted version 0 in
term 0
[2019-09-03T20:11:14,786][INFO ][o.e.c.c.JoinHelper ] [elastic-kibana] failed to join
{elastic02}{FT0SSbtQQkOvIoh7qwzvYg}{ooFyk1dqQ36hNK7CojUABw}{10.2.208.27}
{10.2.208.27:9300}{ml.machine_memory=33742123008, ml.max_open_jobs=20,
xpack.installed=true} with JoinRequest{sourceNode={elastic-kibana}
{sWb4AwzaRPGWmxJYCo3Sgw}{svrZlwvJSeC2J2P6RYmRwQ}{10.4.28.35}{10.4.28.35:9300}
{ml.machine_memory=8375558144, xpack.installed=true, ml.max_open_jobs=20},
optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [elastic02][10.2.208.27:9300]
[internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [elastic-kibana]
[10.4.28.35:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1306) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
[2019-09-03T20:11:14,787][INFO ][o.e.c.c.JoinHelper ] [elastic-kibana] failed to join
{elastic02}{FT0SSbtQQkOvIoh7qwzvYg}{ooFyk1dqQ36hNK7CojUABw}{10.2.208.27}
{10.2.208.27:9300}{ml.machine_memory=33742123008, ml.max_open_jobs=20,
xpack.installed=true} with JoinRequest{sourceNode={elastic-kibana}
{sWb4AwzaRPGWmxJYCo3Sgw}{svrZlwvJSeC2J2P6RYmRwQ}{10.4.28.35}{10.4.28.35:9300}
{ml.machine_memory=8375558144, xpack.installed=true, ml.max_open_jobs=20},
optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [elastic02][10.2.208.27:9300]
[internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [elastic-kibana][10.4.28.35:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1306) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
and configured the first 3 nodes, now after a few weeks when I'm trying to add a new node (as coordinator to install kibana in there) Im unable to make it join the cluster
And I also made the cert for this host without dns and ip option but with same results.
There is one more thing when Im creating the cert and this are some warnings
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG
(file:/usr/share/elasticsearch/lib/tools/security-cli/bcprov-jdk15on-1.61.jar) to constructor
sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of
org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access
operations
WARNING: All illegal access operations will be denied in a future release
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.
I would start with a basic telnet from the node you want to join to the nodes that are already there. On port 9200 and on 9300 if you get a connected then start looking at your config, but what I can tell from the logs the node who wants to join cannot connect to the existing nodes.
I made sure I have same uid/gid for elasticsearch/logstash/kibana
install all rpm
copy all config file from master
change name/ip on each config file
copy .p12 key from master
and started new node's elasticsearch.service and it join the cluster.
And then I've copied the files to /etc/elasticsearch/certs with elasticsearch:elasticsearch as owner
And as I told you the cluster has formed with the first 3 nodes
On the master node I see this
[2019-09-04T18:52:18,831][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [elastic02] client did
not trust this server's certificate, closing connection
Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.4.28.35:51000
Finally Im seeing some light at the end of the tunnel I was having a issue with the security group, but then again Im still facing a problem
At the current master I see:
[2019-09-05T18:03:04,292][WARN ][o.e.t.TcpTransport ] [elastic01] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:58554, remoteAddress=10.4.28.35/10.4.28.35:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
....
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
....
Caused by: java.security.cert.CertificateException: No subject alternative names present
At the "4th" node I see:
[2019-09-05T18:14:21,484][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [elastic-kibana] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.2.208.26:60140}
In addition Im not understanding why the 3 firsts nodes have joined and this does not. I've created the certs in the same way for all the 4 nodes
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.