Unable to enable https and internode communication in 2 node cluster

Elastic Stack Elasticsearch
1

Hi,

I have setup a 2 node ES cluster as follows
node-1 : elasticsearch and kibana
node-2: elasticsearch

I have enabled https and tls level security on node-1 successfully on both between kibana and browser and between kibana and elasticsearch. However

  1. When I try to add https to node-2 it is asking me for username and password.
  2. Unable to make it join the cluster with node-1

I have generated a organisation level certificate in .pfx format for node-1 and using the same for node-2. Is this an issue? Should there be a new certificate issued for node-2?

Following is my es config from node-1 and node-2:
node-1:

cluster.name: cluster1
node.name: node-1
path.data: E:/elasticsearch/data
path.logs: E:/elasticsearch/logs
network.host: node-1

discovery.seed_hosts: ["node-2", "node-1"]
cluster.initial_master_nodes: ["node-1"]
node.master: true
node.data: true

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.http.ssl.truststore.path: <path-to-file-elk.pfx> 

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.transport.ssl.truststore.path: <path-to-file-elk.pfx>

node-2:

cluster.name: cluster1
node.name: node-2
path.data: E:/elasticsearch/data
path.logs: E:/elasticsearch/logs
network.host: node-2
discovery.seed_hosts: ["node-2", "node-1"]
cluster.initial_master_nodes: ["node-1"]
node.master: true
node.data: true

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.http.ssl.truststore.path: <path-to-file-elk.pfx>

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: <path-to-file-elk.pfx>
xpack.security.transport.ssl.truststore.path: <path-to-file-elk.pfx>

Could someone kindly guide me on this?

1 Like
2

Can you clarify what you mean with "try to add https" and what is asking you for a username and a password and at which point ?

3

Hi @ikakavas, thanks for looking into this :slight_smile:
I mean when I try to enable https on elasticsearch on node-2 (from http://node-2:9200 to https://node-2:9200). I am being asked for username and password on accessing https://node-2:9200.

The ES logs on this node is also showing the following error in logs :
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate

4

Could someone pls help me out here? @ikakavas @Magnus_Kessler @magnusbaeck

5

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

6

Apologies for the pings.

Kindly look into my query when you have the time.

7

You need a separate certificate for node2.

Here's an excellent guide that I advise you to follow through:

Note that the author uses separate certificate/key files and not keystores, IMO it is easier to follow his approach, keystore and truststore concepts are harder to grasp for most users/admins/developers..

1 Like
8

@hunsw - I have followed this blog and it says You can use the scp command to copy certificates from node1 to node2. Both nodes require the certificate and key in order to secure the connection.

So does that mean a SINGLE ssl certificate would have domain names of both node-1 and node-2. I am totally new to network security, hence pardon my silly questions.

Also can we use this certificate then on the logstash nodes as well since we would have to enable security there too?

9

It's ok. In the guide he uses node1 to generate certificates.

After the generation step, the certificate and key of node2 and the certificate authority (CA) certificate has to be copied (with scp for example) from node1 to node2.

10

You only need the CA certificate on Logstash, so your Logstash nodes can be sure that the certificate of node1 and node2 are issued by a trusted authority.

EDIT: unless you want mutual authentication (with client cert).

11

Ok so there will be 2 separate files - node1.cer and node2.cer?

Also regarding CA, we can download this from the browser since https for kibana is working ? @hunsw

For mutual auth - CA and client cert both are needed, correct? :slight_smile:

12

Yes, you can download it from the browser too, though it has to be on your nodes too somewhere. (Probably in the truststore file.)

Yes, for mutual auth you need the CA and the client cert too. Or you can just use a user/password combination for Logstash to authenticate in Elasticsearch and secure the channel with TLS.

See here:
https://www.elastic.co/guide/en/logstash/current/ls-security.html

13

Great! I will request for a cert for the other node and get back to you in case of any queries.

This is how my logstash output conf looks as of now, kindly let me know if you see anything wrong in this @hunsw

output {
elasticsearch {
	user => logstash_internal
	password => kibana
	ssl => true
	cacert => "e:/logstash/config/ca-cert.cer" #downloaded from the browser
	hosts => ["https://node1:9200/","https://node2:9200/"]
    index => "ls-%{+YYYY.MM.dd}"	
}
}
14

Looks good to me, hopefully the cacert path is a valid one. (I haven't run LS on Windows, but shouldn't that be e:\logstash\config\ca-cert.cer?)

15

I will re-check that :slight_smile:
Thank you for your time! Appreciate it!

16

Hi @hunsw - the setup is working fine from logstash, thank you for your help!

I am facing issues currently with nodes unable to form a cluster after enabling security and generating node1.pfx and node2.pfx

[node1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=node2...] [DNS:node2]; .......... certificate is not trusted in this ssl context ([xpack.security.transport.ssl])
17

@ikakavas - The config I am using is mentioned in this mail. I cannot mention the node names and other organisation specific details as it against the policy. If you need any information pls let me know.

The certificates have been generated by my organisation and both the node certificates have the same CA (i have confirmed this) , the certificates are in .pfx format.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.