Unable to enable https and internode communication in 2 node cluster

nityaraj06 · May 12, 2020, 7:31pm

Hi,

I have setup a 2 node ES cluster as follows
node-1 : elasticsearch and kibana
node-2: elasticsearch

I have enabled https and tls level security on node-1 successfully on both between kibana and browser and between kibana and elasticsearch. However

When I try to add https to node-2 it is asking me for username and password.
Unable to make it join the cluster with node-1

I have generated a organisation level certificate in .pfx format for node-1 and using the same for node-2. Is this an issue? Should there be a new certificate issued for node-2?

Following is my es config from node-1 and node-2:
node-1:

cluster.name: cluster1
node.name: node-1
path.data: E:/elasticsearch/data
path.logs: E:/elasticsearch/logs
network.host: node-1

discovery.seed_hosts: ["node-2", "node-1"]
cluster.initial_master_nodes: ["node-1"]
node.master: true
node.data: true

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.http.ssl.truststore.path: <path-to-file-elk.pfx> 

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.transport.ssl.truststore.path: <path-to-file-elk.pfx>

node-2:

cluster.name: cluster1
node.name: node-2
path.data: E:/elasticsearch/data
path.logs: E:/elasticsearch/logs
network.host: node-2
discovery.seed_hosts: ["node-2", "node-1"]
cluster.initial_master_nodes: ["node-1"]
node.master: true
node.data: true

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.http.ssl.truststore.path: <path-to-file-elk.pfx>

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.transport.ssl.truststore.path: <path-to-file-elk.pfx>

Could someone kindly guide me on this?

ikakavas · May 13, 2020, 6:15am

Can you clarify what you mean with "try to add https" and what is asking you for a username and a password and at which point ?

nityaraj06 · May 13, 2020, 6:35am

Hi @ikakavas, thanks for looking into this
I mean when I try to enable https on elasticsearch on node-2 (from http://node-2:9200 to https://node-2:9200). I am being asked for username and password on accessing https://node-2:9200.

The ES logs on this node is also showing the following error in logs :
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate

nityaraj06 · May 13, 2020, 2:31pm

Could someone pls help me out here? @ikakavas @Magnus_Kessler @magnusbaeck

ikakavas · May 14, 2020, 8:38am

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

nityaraj06 · May 14, 2020, 8:55am

Apologies for the pings.

Kindly look into my query when you have the time.

hunsw · May 14, 2020, 12:16pm

You need a separate certificate for node2.

Here's an excellent guide that I advise you to follow through:

Note that the author uses separate certificate/key files and not keystores, IMO it is easier to follow his approach, keystore and truststore concepts are harder to grasp for most users/admins/developers..

nityaraj06 · May 14, 2020, 12:34pm

@hunsw - I have followed this blog and it says You can use the scp command to copy certificates from node1 to node2. Both nodes require the certificate and key in order to secure the connection.

So does that mean a SINGLE ssl certificate would have domain names of both node-1 and node-2. I am totally new to network security, hence pardon my silly questions.

Also can we use this certificate then on the logstash nodes as well since we would have to enable security there too?

hunsw · May 14, 2020, 12:38pm

It's ok. In the guide he uses node1 to generate certificates.

After the generation step, the certificate and key of node2 and the certificate authority (CA) certificate has to be copied (with scp for example) from node1 to node2.

hunsw · May 14, 2020, 12:39pm

You only need the CA certificate on Logstash, so your Logstash nodes can be sure that the certificate of node1 and node2 are issued by a trusted authority.

EDIT: unless you want mutual authentication (with client cert).

nityaraj06 · May 14, 2020, 12:52pm

Ok so there will be 2 separate files - node1.cer and node2.cer?

Also regarding CA, we can download this from the browser since https for kibana is working ? @hunsw

For mutual auth - CA and client cert both are needed, correct?

hunsw · May 14, 2020, 1:03pm

Yes, you can download it from the browser too, though it has to be on your nodes too somewhere. (Probably in the truststore file.)

Yes, for mutual auth you need the CA and the client cert too. Or you can just use a user/password combination for Logstash to authenticate in Elasticsearch and secure the channel with TLS.

See here:
https://www.elastic.co/guide/en/logstash/current/ls-security.html

nityaraj06 · May 14, 2020, 1:07pm

Great! I will request for a cert for the other node and get back to you in case of any queries.

This is how my logstash output conf looks as of now, kindly let me know if you see anything wrong in this @hunsw

output {
elasticsearch {
	user => logstash_internal
	password => kibana
	ssl => true
	cacert => "e:/logstash/config/ca-cert.cer" #downloaded from the browser
	hosts => ["https://node1:9200/","https://node2:9200/"]
    index => "ls-%{+YYYY.MM.dd}"	
}
}

hunsw · May 14, 2020, 1:13pm

Looks good to me, hopefully the cacert path is a valid one. (I haven't run LS on Windows, but shouldn't that be e:\logstash\config\ca-cert.cer?)

nityaraj06 · May 14, 2020, 1:17pm

I will re-check that
Thank you for your time! Appreciate it!

nityaraj06 · May 19, 2020, 10:08am

Hi @hunsw - the setup is working fine from logstash, thank you for your help!

I am facing issues currently with nodes unable to form a cluster after enabling security and generating node1.pfx and node2.pfx

[node1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=node2...] [DNS:node2]; .......... certificate is not trusted in this ssl context ([xpack.security.transport.ssl])

nityaraj06 · May 21, 2020, 7:50am

@ikakavas - The config I am using is mentioned in this mail. I cannot mention the node names and other organisation specific details as it against the policy. If you need any information pls let me know.

The certificates have been generated by my organisation and both the node certificates have the same CA (i have confirmed this) , the certificates are in .pfx format.

system · June 18, 2020, 7:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.