Elasticsearch certificate error when access with public ip

Hi,

Problem: Elasticsearch cluster url having a certificate error, needed this to configure on beats on other vm.

I am experimenting it on Azure VM with ubuntu operating system.

Followed this official guide to install Elasticsearch

Install Elasticsearch with Debian Package | Elasticsearch Guide [8.4] | Elastic

Cluster url's, with the localhost, hostname and with the private ip address are working but not with public ip.

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://depf.elk-demo.com:9200

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://10.0.0.5:9200

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://public-ip-address:9200

I am getting the certification error using public ip, the reason is certificate not containing public ip as part of subject alternative names.

root@depf:/home/depf# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://public-ip-address:9200
Enter host password for user 'elastic':
curl: (60) SSL: no alternative certificate subject name matches target host name ''

could you please suggest, where to tweak to make it work?

the following is the elasticsearch.yml

root@depf:/home# grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["depf.elk-demo.com"]
http.host: [local, site]
root@depf:/home#

Hi @jay.annapureddy Welcome to the community.

So you will need to regenerate the HTTP cert to include the subject names / IPs you want

see Here

run this command and answer the questions include all the local and public IPs and FQDN

bin/elasticsearch-certutil http

or you can use and instances.yml and pass it in.

Then set the path to your new cert or replace the old one.

When you do the grep thing on your .yml it takes out the indenting which makes it hard to understand

xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12 <!--- Your New Cert Here

Hi @stephenb thank you for quick response,

I followed your instructions to generate new certificate, following the output.

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil http

Generate a CSR? [y/N]N

Use an existing CA? [y/N]N

Subject DN: CN=Elasticsearch HTTP CA
Validity: 5y
Key Size: 2048

Do you wish to change any of these options? [y/N]N

CA password:  [<ENTER> for none]
Repeat password to confirm: 

For how long should your certificate be valid? [5y] 90D

Generate a certificate per node? [y/N]N

You entered the following hostnames.

 - localhost
 - depf.bel.com
 - depfvm2.bel.com

Is this correct [Y/n]Y

You entered the following IP addresses.

 - 20.105.249.142
 - 10.0.0.5
 - 127.0.0.1

Is this correct [Y/n]Y


Key Name: localhost
Subject DN: CN=localhost
Key Size: 2048

Do you wish to change any of these options? [y/N]N

Provide a password for the "http.p12" file:  [<ENTER> for none]
Repeat password to confirm: 

Zip file written to /etc/elasticsearch/certs/depf.zip

unzip -q depf.zip -d /etc/elasticsearch/certs/depf/

sudo nano /etc/elasticsearch/elasticsearch.yml

cat /etc/elasticsearch/elasticsearch.yml

root@depf:/etc/elasticsearch/certs/depf/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-keystore add "xpack.security.http.ssl.keystore.secure_password"
Setting xpack.security.http.ssl.keystore.secure_password already exists. Overwrite? [y/N]y
Enter value for xpack.security.http.ssl.keystore.secure_password: 
root@depf:/etc/elasticsearch/certs/depf/elasticsearch# 

and below is the final configuration referring to new certificate

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 28-09-2022 06:49:06
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/depf/elasticsearch/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["depf.bel.com"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

still I am having the error(different one), where did it went wrong ? could you please suggests?


root@depf:/etc/elasticsearch/certs/depf/elasticsearch# sudo systemctl restart elasticsearch.service
root@depf:/etc/elasticsearch/certs/depf/elasticsearch# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@depf:/etc/elasticsearch/certs/depf/elasticsearch# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://20.105.249.142:9200
Enter host password for user 'elastic':
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@depf:/etc/elasticsearch/certs/depf/elasticsearch#

What OS are you on?

Did you actually look at the page above? ... depending on your specific OS, self signed certs can be very painful / require additional configuration.

It is Azure VM Linux (ubuntu 20.04)

I am not well understood the content of curl - SSL CA Certificates and also I get certificate error from browser https://20.105.249.142:9200

cluster is up and running as I see it works with curl -k

also I generated certs from windows pc and copied to server and tried, same result. My doubts on the generation utility, could you please advice ?

PS C:\elasticsearch-8.4.2\certs> C:/elasticsearch-8.4.2/bin/elasticsearch-certutil ca --out C:/elasticsearch-8.4.2/certs/elastic-stack-ca.p12 --pass pathfinders

C:/elasticsearch-8.4.2/bin/elasticsearch-certutil cert --in C:/elasticsearch-8.4.2/certs/instances.yml --out C:/elasticsearch-8.4.2/certs/depf.zip --pass pathfinders --ca C:/elasticsearch-8.4.2/certs/elastic-stack-ca.p12

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 29-09-2022 07:58:29
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/depf-es-node1.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["depf.bel.com"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
Last login: Thu Sep 29 10:06:30 2022 from 13.81.37.99
depf@depf:~$ sudo su
root@depf:/home/depf# sudo nano /etc/elasticsearch/elasticsearch.yml 
root@depf:/home/depf# sudo systemctl restart elasticsearch.service
root@depf:/home/depf# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@depf:/home/depf# curl -k --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
{
  "name" : "depf.bel.com",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "7-DD4XTgTuKMl4dZSBljVg",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
root@depf:/home/depf#

Certainly creating it on another server and moving it over would not work.

You can also try curl with the -v without -k And it will give you more details on the certificate errors.

Assuming you're running vanilla Ubuntu You should not be having this problem so I think somehow you're still not generating the certain correct?.

Try the -v and see what it tells you

This is the image name used while creating Azure VM
Ubuntu Server 20.04 LTS - Gen2

root@depf:/etc/elasticsearch/certs# curl -k --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
{
  "name" : "depf.bel.com",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "s_FOTL1QSAeyw67UsEivGw",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
root@depf:/etc/elasticsearch/certs# curl -v --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
*   Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/elasticsearch/certs/http_ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@depf:/etc/elasticsearch/certs#

I am recreating VM and will try installing again to see if error persists

No luck with new VM as well,

Here is the full story

Created a Azure VM Linux (ubuntu 20.04) Image name: Ubuntu Server 20.04 LTS - Gen2

Installed Elasticsearch by referencing Install Elasticsearch with Debian Package | Elasticsearch Guide [8.4] | Elastic

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

sudo apt-get install apt-transport-https

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt-get update && sudo apt-get install elasticsearch

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Successfully started, able to access the localhost:9200

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

but with public ip, got the certificate error, see below

root@depfVM2:/home/depf# curl -k --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://13.94.129.124:9200
Enter host password for user 'elastic':
{
  "name" : "depfVM2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "y-iPnRvdRG6I087UkNXLTg",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
root@depfVM2:/home/depf# curl -v --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://13.94.129.124:9200
Enter host password for user 'elastic':
*   Trying 13.94.129.124:9200...
* TCP_NODELAY set
* Connected to 13.94.129.124 (13.94.129.124) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/elasticsearch/certs/http_ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=depfVM2
*  start date: Sep 29 15:20:43 2022 GMT
*  expire date: Sep 28 15:20:43 2024 GMT
*  subjectAltName does not match 13.94.129.124
* SSL: no alternative certificate subject name matches target host name '13.94.129.124'
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name '13.94.129.124'
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@depfVM2:/home/depf#

Now trying to create a certificate with public ip

created a instances.yml file

instances:
  - name: "node1" 
    ip: 
      - "13.94.129.124"
    dns: 
      - "localhost"

certificate generated (** in the below, password is masked to you **)

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/certs/elastic-stack-ca.p12 --pass ********

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in /etc/elasticsearch/certs/instances.yml --out /etc/elasticsearch/certs/depf.zip --pass ******* --ca /etc/elasticsearch/certs/elastic-stack-ca.p12

unzip -q depf.zip -d /etc/elasticsearch/certs/depf/

sudo nano /etc/elasticsearch/elasticsearch.yml

grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml

/usr/share/elasticsearch/bin/elasticsearch-keystore add "xpack.security.http.ssl.keystore.secure_password"

sudo systemctl restart elasticsearch.service

now with created certificate, neither localhost nor public ip is working, see below

root@depfVM2:/etc/elasticsearch/certs/depf/node1# curl -k --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
{
  "name" : "depfVM2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "y-iPnRvdRG6I087UkNXLTg",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
root@depfVM2:/etc/elasticsearch/certs/depf/node1# curl -v --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
Enter host password for user 'elastic':
*   Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/elasticsearch/certs/http_ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@depfVM2:/etc/elasticsearch/certs/depf/node1# 

/etc/elasticsearch/elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 29-09-2022 15:20:35
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/depf/node1/node1.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["depfVM2"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Cloud you please see where it went wrong and suggest for the correct action, thanks a lot in advance

Not sure I'll be able to take a look at this for a day or two.

Where are you doing the curl from? the Same Server?

Can you try the Curl from another Server or your desktop?

I got the same issue following your directions....

Here is your issue... subtle ... easy to make.. took me a bit to figure

You do this... Good!

Then you do this good!! ... but I would put localhost in the dns and 127.0.0.1 and whatever the local IP is as well so all work..

My instances.yml looks like

instances:
  - name: "stephenb-test-vm" 
    ip: 
      - "127.0.0.1"
      - "10.0.0.4"
      - "20.3.xxx.xxx"
    dns: 
      - "stephenb-test-vm"
      - "localhost"

Now here is the catch ... you are then trying to use curl with the wrong CA...
Bad! :slight_smile: Won't work
--cacert /etc/elasticsearch/certs/http_ca.crt

That is NOT the CA you generated the new certs from... won't work!

AND you can't use the new elastic-stack-ca.p12 .. well because its a .p12 you need to convert it to a .crt or .pem.

if you try to use the .p12 you will get

* error setting certificate verify locations:
  CAfile: /etc/elasticsearch/certs/new/elastic-stack-ca.p12
  CApath: /etc/ssl/certs

So you need to get a .pem ...

see here

openssl pkcs12 -in elastic-stack-ca.p12 -out elastic-stack-ca-cert.pem -clcerts -nokeys

Now you can use that as the ca file

--cacert /etc/elasticsearch/certs/elastic-stack-ca-cert.pem

That should work ... YUP SSL Cert stuff ... arrggghhh it always gets me too!

This then worked, via my laptop! to the Azure VM / Elasticsearch

hyperion:azure sbrown$ curl -v -u elastic --cacert ./elastic-stack-ca-cert.pem https://20.3.xxx.xxx:9200

Enter host password for user 'elastic':
*   Trying 20.3.xxx.xxx:9200...
* Connected to 20.3.xxx.xxx (20.3.xxx.xxx) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: ./elastic-stack-ca-cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=stephenb-test-vm
*  start date: Sep 30 00:13:07 2022 GMT
*  expire date: Sep 29 00:13:07 2025 GMT
*  subjectAltName: host "20.3.xxx.xxx matched cert's IP address!
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: 20.3.226.205:9200
> Authorization: Basic ZWxhc3RpYzpHYmhHTXR1MWpKS0ZyNFY3TUNJaw==
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 539
< 
{
  "name" : "stephenb-test-vm",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ytMhbiQ4QbShIKce-EPAOw",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host 20.3.xxx.xxx left intact

Note I did not do all the password stuff you should just be careful....

Thank you @stephenb, indeed your suggestion worked for me from vm, below is output
but from laptop it did not work same from browser too https://13.94.129.124:9200

Output from VM

root@depfVM2:/etc/elasticsearch/certs# curl --cacert /etc/elasticsearch/certs/elastic-stack-ca-cert.pem -u elastic https://localhost:9200
Enter host password for user 'elastic':
{
  "name" : "depfVM2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "y-iPnRvdRG6I087UkNXLTg",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

root@depfVM2:/etc/elasticsearch/certs# curl -v --cacert /etc/elasticsearch/certs/elastic-stack-ca-cert.pem -u elastic https://13.94.129.124:9200
Enter host password for user 'elastic':
*   Trying 13.94.129.124:9200...
* TCP_NODELAY set
* Connected to 13.94.129.124 (13.94.129.124) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/elasticsearch/certs/elastic-stack-ca-cert.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=depfVM2
*  start date: Sep 30 06:55:57 2022 GMT
*  expire date: Sep 29 06:55:57 2025 GMT
*  subjectAltName: host "13.94.129.124" matched cert's IP address!
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: 13.94.129.124:9200
> Authorization: Basic ZWxhc3RpYzpzMkhBYUYwUGZDbmxnRGVTSy1MVQ==
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 530
< 
{
  "name" : "depfVM2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "y-iPnRvdRG6I087UkNXLTg",
  "version" : {
    "number" : "8.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "89f8c6d8429db93b816403ee75e5c270b43a940a",
    "build_date" : "2022-09-14T16:26:04.382547801Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host 13.94.129.124 left intact

Output from laptop

C:\Users\jayan>curl --cacert C:/elasticsearch-8.4.2/certs/elastic-stack-ca-cert.pem -u elastic https://13.94.129.124:9200
Enter host password for user 'elastic':
curl: (60) schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

C:\Users\jayan>curl -v --cacert C:/elasticsearch-8.4.2/certs/elastic-stack-ca-cert.pem -u elastic https://13.94.129.124:9200
Enter host password for user 'elastic':
*   Trying 13.94.129.124:9200...
* Connected to 13.94.129.124 (13.94.129.124) port 9200 (#0)
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* ALPN: offers http/1.1
* schannel: added 1 certificate(s) from CA file 'C:/elasticsearch-8.4.2/certs/elastic-stack-ca-cert.pem'
* schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN
* Closing connection 0
curl: (60) schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

C:\Users\jayan>

when I check host and port from Check SSL Certificate

Could you please advice ?

In think That is a windows issue

perhaps run curl --version

From the link curl - SSL CA Certificates

Certificate Verification with Schannel and Secure Transport

If libcurl was built with Schannel (Microsoft's native TLS engine) or Secure Transport (Apple's native TLS engine) support, then libcurl will still perform peer certificate verification, but instead of using a CA cert bundle, it will use the certificates that are built into the OS. These are the same certificates that appear in the Internet Options control panel (under Windows) or Keychain Access application (under OS X). Any custom security rules for certificates will be honored.

So you might need to install the elastic CA. Not my area of expertise on Windows.

You might open a separate thread with a very specific subject line like

Help getting windows curl to accept Elastic CA

And then provide the details.

Thank you @stephenb, I will do as you suggested.

By the way any idea why public ip of Azure VM is not part of initial certificate generated during the installation of elasticsearch ?
My problem will solve if it includes in the first place, any thoughts?

Yeah I have an idea but you should validate it with you network experts

... the Public IP is NATed (translated at a gateway)... meaning it does not actually exist on the VM so the local VM does not really "know" its external IP.

if you run ifconfig that external IP won't show up.... therefor the local tools like certgen are not aware of the external IP.

So you need to add it manually or through some automation

example on my VM, note the VM only knows about its local addresses because the gateway is what does the translation from your Public IP to Private IP

azureuser@stephenb-es-test:~$ ifconfig
enP9043s1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 00:0d:3a:fc:db:a4  txqueuelen 1000  (Ethernet)
        RX packets 917  bytes 1059991 (1.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1820  bytes 381478 (381.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.4  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::20d:3aff:fefc:dba4  prefixlen 64  scopeid 0x20<link>
        ether 00:0d:3a:fc:db:a4  txqueuelen 1000  (Ethernet)
        RX packets 2395  bytes 2236899 (2.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1770  bytes 377758 (377.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 98  bytes 10906 (10.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 98  bytes 10906 (10.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.