Unable to start ES 7.3.1 after configuring internode communication encryption

gdamico · September 11, 2019, 9:57pm

Hi,

After completing "Getting started with the Elastic Stack" and "Securing the Elastic Stack" tutorials, I'm trying to complete "Encrypting Communications". After CA and certificate creation, Keystore and Truststore passwords addittion, and elasticsearch.yml configuration, ES is unable to start. I'm getting the "keystore password was incorrect" error in the service log.

I've already read these posts, but can't find a solution :

Problem with keystore password was incorrect
Elasticsearch 7.1.x fails to start with security enabled

I did these actions:

Deleted CA and certificate files and created again tu ensure password correctness.
Overwriten elasticsearch-keystore and added keystore and truststore passwords to ensure correctness.
elasticsearch.yml edition to ensure parameters correctness.

Installation properties:

Ubuntu 19.04 virtual machine
DEB package installation mode
Kibana, Metricbeat, and Logstash installed with authentication enabled

The Service log:

Caused by: java.io.IOException: keystore password was incorrect
	at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2117) ~[?:?]
	at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222) ~[?:?]
	at java.security.KeyStore.load(KeyStore.java:1472) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:89) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createTrustManager(StoreKeyConfig.java:83) ~[?:?]
	at

The configuration file:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: test-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
#discovery.type: single-node
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/node-1.p12
xpack.security.transport.ssl.keystore.truststore.path: /etc/elasticsearch/certs/node-1.p12

I'd really appreciate your help.

Regards

ikakavas · September 12, 2019, 9:47am

is self explanatory.

is wrong, it should be

xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/node-1.p12

How ? What was the setting name ? Can you share the output of

bin/elasticsearch-keystore list

?

You can verify the password outside of elasticsearch running

openssl pkcs12 -in /etc/elasticsearch/certs/node-1.p12

it will prompt you for the password, so you can try and see if it is the password you think it is.

gdamico · September 12, 2019, 2:31pm

Hi, Loannis!

Thanks for your reply. I´m going to answer yor questions one by one:

This was corrected, tryed to start ES again, but got the same behaviour:

This is what I did to overwrite de Keystore:
./bin/elasticsearch-keystore create
The output you requested:
root@gian-sandbox-vm:/home/gian# /usr/share/elasticsearch/bin/elasticsearch-keystore list
keystore.seed
xpack.security.ssl.keystore.secure_password
xpack.security.ssl.truststore.secure_password
root@gian-sandbox-vm:/home/gian#
The output you requested

ikakavas:

You can verify the password outside of elasticsearch running
openssl pkcs12 -in /etc/elasticsearch/certs/node-1.p12
it will prompt you for the password, so you can try and see if it is the password you think it is.

`root@gian-sandbox-vm:/home/gian# openssl pkcs12 -in /etc/elasticsearch/certs/node-1.p12
Enter Import Password:
Bag Attributes
    friendlyName: instance
    localKeyID: 54 69 6D 65 20 31 35 36 38 32 32 38 37 32 38 31 34 32 
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFHDBOBgkqhkiG9w0BBQ0wQTApBgkqhkiG9w0BBQwwHAQI+949tGCRPR0CAggA
MAwGCCqGSIb3DQIJBQAwFAYIKoZIhvcNAwcECIP2cOP9xrh3BIIEyN8vDfH3rvhu
Eyr4WrwuWhFk9a5L95Ge++6kYDWYdRPHovm77Vqh+nR5kFUklsNgrZf/qcdBghgo
Fl1P9q6LNjRLOTWjclfdePdta9b8wY4rbhoTpq2N/5jtb6C0J1sFqYzPwS84I4f3
Qe03mnZfAf0XiAd1tGXuR7JQLQaSzmwK4mQ4oNlnTcpbZ7D6tEPUFov0Fxq2Xeac
pSiuEoiWnrvR+qX8vejc2sPKPuihbG6E8oAA0bytMJsi18o3NngAk1p4O62X01JY
I8359qZwb3mgj/g9YuM/0yUQ4TQdtStN/vRVBNpUR5hA27r4/RJtaFWdLXBTJG/M
fmAcmuKU01dvEtSQb5Zp4hU1NbCyIvBpkXnEBErK1QycDtSl65P3itbBh4QRY4M+
qMUQzU9c9K3OckihFYpOETg9EyS6tFYS+FK5xZ5U35fgwUEjGzc17XpOfbalFrjo
FwbV/cTePqtPWBl1QHbn5DvvI/YqFi87YvBZcBKPU2txrGRKRekMc7pCtJwMh3sy
qCf3TzwzmRHk/WYZoXDud2b1mN/DKrURdWHorVuP95yWv3Vz+8kdYDas2Mb3/ahp
0Dl35Eyi1KdA6zN6fr0pMxpdUibRLp2LwCqVg2Exgb9AvuLxWStu4NBn24GrnVCa
Wnm1fiTJ5gARB2S9WsX3Grx3d3t0EXWI+AzDTWUFWgotVwjseupcWWmvxkQuYe8F
xtqh7KAohZW1Am3IupLDoZd4vnce7xUTUgFkhePtMIRUkcdByaPkIOPha3fHJIri
o37X3eu8cZ9v2wHrzxnHFley1a0JBMOEmAgHka4shHoX8j5KPF/YAY95TFLg5c+1
wF/5Uad9Tzr2mRaKiQXdQYnXYlk15Pxn5sK7sD/fkuosFATTYG/Z2Nu7fgrL8SUJ
gR2aopYnlnQ1mURJWMrTy+1FW+WWFhRwxyPdB0tQ6Y1ThhL+tV1GY/vmLzxMscEu
vnvBPo4anUGpk3znnOA4FF62P6bQ6uzmEBEd8z9m7aqzdR0GwJ4ML5MaE5JM6oYm
QNo9GXH45WGg1+zBerUMuoJKIQKDpIyhoIcRcVnRFsyC8ithScCLbXHRsDvspgLG
EHfxT+UtGeDPs4Cjgcu+3LJtDohBUt0PLWJ1Pf78bAMtLndRO1Oh0lAItNecD1dZ
bvVQvjFIjdI4ko1tnkFbilgPeGQMvW1Nue9N0kDgxbj0vcoaBb/BPx/6mffJVYYm
2TFYdLo3HmHbCzraiXf8TRKC344BiUamrr1kg0l/aiVn8LtrrrEDrUkL0c3kL6/k
Lz3EJzPnqW9AC+UioqadOp6P6EHBRTcVm+x/3c7v+l8uwb3Jw8ShxNItABNItrO/
JTwf1maSI+PEs0vQhN7BGo9Wlx1yQkYeo8oiTKUjlt9bGtIycbJaPNlSD/Gl4wu8
x7k1OfSSVv4to5ncyxkFMvYBz7mNJtKqvrprW2m+KdOri71x0AgYq2gWnmqTA/mQ
XMg098VDwAq98Afmt+kdZZw0GyoEFJFDxgHbM2xdkhPoevGxHqKee3loOsYix5yZ
TgAQ3vkhm8SRBCB+n7Uw5dJKEKUNC3kDQll6o+iUfoqgk+ETP18vdlg+zwEoK4PP
xFXUUcTj9YT1Ob8Z+zoXjQ==
-----END ENCRYPTED PRIVATE KEY-----
Bag Attributes
    friendlyName: instance
    localKeyID: 54 69 6D 65 20 31 35 36 38 32 32 38 37 32 38 31 34 32 
subject=CN = instance

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIVAO3ZNUC2GQxVT+uJQDsUQy5GNkXOMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTE5MDkxMTE5MDUxN1oXDTIyMDkxMDE5MDUxN1owEzERMA8G
A1UEAxMIaW5zdGFuY2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDk
BhtLVgSibyWluj1iOUq5gQJr0qfUhwd2/wE6jrDqyF9UOZK8SN8t+A/Ypbr5uAlX
2Kwd32jWDp12/0CK//zVEE7D2Ahln7ivcX9mreCgax15fOzejc6tqvTjsYFp4MVl
Kta+YbXGls+U5/A28JBHRVPQUtvjoFtjcE+PV3XAjSP1cXsdH8JNZObPC/VJ5xQJ
f1v1AyYyBQCr3zbpLuxEz7p+m/OQ8Dl1c/a8eAOn6lT3kaEMk3QhSXVeqEE9pHpZ
LAMn2w7fEAYmFzo+kKinfK0gFpDsiblhLShFQZL3rk9ek0yfMFd+YjURiU16RNzb
LOv7Mlsb9lCkUJ1USr5NAgMBAAGjezB5MB0GA1UdDgQWBBQ9a32rwDTxKRiZyiVS
XVqkvH5IQDAfBgNVHSMEGDAWgBSaGWrwangzH8xaBcrQu+kMKWp/1DAsBgNVHREE
JTAjgglsb2NhbGhvc3SHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCQYDVR0TBAIw
ADANBgkqhkiG9w0BAQsFAAOCAQEACOVSAKzclOdWRH2meZVtJtYBxdgf06kIgfJa
S+mS2/eE7W6TSfulDS0cAzf/EGWrDUOJvvLHc0hrJr2/rxLLqR0uEx8azzc917/J
FR9ZdnxN84KKhxyejz1K4ZJbGWXWxhbeb2oIQsimozt2czlt9ly9F5YTbCdiJFg+
rTvCR59MWbPufdcuniU8lvrspBeem6BRWLZ5eEx+yG3XfJ9W7pF7TixQNIft4A5z
JC8saegXJZDzRkCQ1w7BlnzkloDAWMlWtEZe7pRjOg1H5iFm6SfaQyxkuT/GY90R
0hjeyV8uPmVM29ihI1umq1f40GrnXa3G0pIVzkXvzrg/YT687g==
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: ca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=CN = Elastic Certificate Tool Autogenerated CA

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----
MIIDSTCCAjGgAwIBAgIUNCrB+ayuy0l06zz8sjwfuwN4DcEwDQYJKoZIhvcNAQEL
BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
cmF0ZWQgQ0EwHhcNMTkwOTExMTkwMzM3WhcNMjIwOTEwMTkwMzM3WjA0MTIwMAYD
VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKbks0CsBUGMkcy8ulI3yAxc
CGiZsEfLsom7LXNIHd4mOoJ661ay05DC8fIkuAV9h+3P9LWWvtugME6gvHLFwg1c
jIXzllpWIqsK4UOam+AjTWGokJEjEWmh8BU889ZdHB77t5qaTfo+/9CgcUewG/9I
h7JKT2PW0AZJ+mpL5iKfyMFNWnaJdZF/5ORyZCsUUFRjNqREEdq72nd80Hz9bRdA
3hU/2deGUlWNnzzg20CjCwHjADaBTIe4vjxyT4LYGy9QxDr2dVbbrjgZCLgRznyp
FT7nKHijxsrqiWLwcoLaXKJOL5b5x63WOuzaez+RRqYGM0BtmEEP9GOfw1mP2CMC
AwEAAaNTMFEwHQYDVR0OBBYEFJoZavBqeDMfzFoFytC76Qwpan/UMB8GA1UdIwQY
MBaAFJoZavBqeDMfzFoFytC76Qwpan/UMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAKN75RWr5B3BhdzvHT/OshN/LpP/3ey6vZOJrJ1WRoSyNnGx
l2k9b1FsBU15p4G4yuAjBiBPaD8j0nMHaZbgQqX2irP3PtiH7liQrudbZC1xgUln
vkEgpDK19HFH9gvHv/IfaAhMBWAlRUdNmdHUprw5U+vkSvx2IjnTZUy6KqeyPp97
h5uKYBqQrTE0fwDd6ULvfpq2ZDiEjY5T/rwnXj+YK4XdZ2lqq3iw+lFlg/JgKys5
54KMCreFvHsOYKSE1kxTjyJ8bydjUiudY4mYihYzsLm0qV2T/8R7n5xUxvMjjg78
MWTDd9S9+lqgYdwINqSTMd6LnojJQb53VZAxA8o=
-----END CERTIFICATE-----
root@gian-sandbox-vm:/home/gian#`

Again, thanks.

ikakavas · September 12, 2019, 2:54pm

That was expected, it was wrong, but not the only error in your config.

You don't have to overwrite the elasticsearch keystore, just set the right settings in it.

This is wrong. The config setting names should be

xpack.security.transport.ssl.truststore.secure_password
xpack.security.transport.ssl.keystore.secure_password

So remove the ones you have with bin/elasticsearch-keystore remove and use bin/elasticsearch-keystore add to add the correct ones.

gdamico · September 12, 2019, 3:35pm

Ok, obviously I did several typo mistakes..
Thanks for your time, Loannis.

system · October 10, 2019, 3:35pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.