Kibana SSL PEM error on Windows


(Matt M) #1

Note: I will regenerate all passwords, certs and ect in my final install since I have provided all that information here.

Environment
Windows 10 Enterprises
Elasticsearch Version : 6.1.1 (elasticsearch-6.1.1.zip)
Kibana Version: 6.1.1 (kibana-6.1.1-windows-x86_64.zip)
Subscriptions : Platinum

Problem:
I am trying to enabled SSL in kibana and I am running into an issue. I cant for the life of me figure out what I have done wrong. Here are the steps I have taken so far.

  1. Create Yaml file use with the certutil --in flag.
instances:
  - name: node1
    dns: ['node1.local']
  - name: devws-kibana
    dns: ['devws-kibana.local']
  1. Declare variables for use in powershell commands
$root = "C:\working\elasticsearch"
[Version]$esVersion = "6.1.1"
$es = "$root\elasticsearch-$($esVersion.ToString())"
$esService = "elasticsearch_$($esVersion.ToString() -replace '\.','')"
[Version]$KibanaVersion = "6.1.1"
$kibana = "$root\kibana-$($KibanaVersion.ToString())-windows-x86_64"
$kibanaService = "elasticsearch-kibana$($KibanaVersion.ToString() -replace '\.','')"
  1. Install x-pack in elasticsearch
`&"$es\bin\elasticsearch-plugin.bat" install x-pack --batch`
  1. Create a Self Signed CA certificate.
 &"$es\bin\x-pack\certutil.bat" ca --silent --pass password --ca-dn "CN=Elasticsearch-DevWS" --pem --out "$root\elastic-stack-ca.zip"
    Expand-Archive -Path "$root\elastic-stack-ca.zip" -OutputPath "$root\certs"
  1. Create a cert for elasticsearch and kibana
&"$es\bin\x-pack\certutil.bat" cert --silent --pem --ca-cert "$root\certs\ca\ca.crt" --ca-key "$root\certs\ca\ca.key" -in "$root\instances.yml" --ca-pass password --pass password --out "$root\certificate-bundle.zip"
Expand-Archive -Path "$root\certificate-bundle.zip" -OutputPath "$root\certs"
  1. Copy Certs to proper directories
Copy-Item -Path "$root\certs\ca\ca.crt" -Destination "$es\config\certs\ca.crt"
Copy-Item -Path "$root\certs\node1\*" -Destination "$es\config\certs\"
Copy-Item -Path "$root\certs\ca\ca.crt" -Destination "$kibana\config\certs\ca.crt"
Copy-Item -Path "$root\certs\devws-kibana\*" -Destination "$kibana\config\certs\"
  1. Update Elasticsearch.yml to below
cluster.name: WRK001
node.name: node1
network.host: node1.local
http.port: 9210
discovery.zen.ping.unicast.hosts: [ 'node1.local' ]
processors: 2
node.master: true
node.data: true
node.max_local_storage_nodes: 1
xpack.ssl.key: certs/node1.key
xpack.ssl.certificate: certs/node1.crt
xpack.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.enabled: true
xpack.ssl.key_passphrase: password
  1. Add secure key passphrase to keystore
"password" |  &"$es\bin\elasticsearch-keystore.bat" add xpack.ssl.secure_key_passphrase --stdin
&"$es\bin\elasticsearch-keystore.bat" list
  1. Start Elasticsearch

  2. Set password for build in accounts

$url = https://node1.local:9210/
$output = & cmd.exe /C "$es\bin\x-pack\setup-passwords.bat auto --url $url -batch" 2>&1
Write-Host -ForegroundColor Green -BackgroundColor Black  $output
  1. Parse passwords from response and save to temp files.
$elasticPassword = ($output | Select-String -Pattern "^PASSWORD\selastic\s=\s(.*)$" -AllMatches).Matches[0].Groups[1].Value
$kibanaPassword = ($output | Select-String -Pattern "^PASSWORD\skibana\s=\s(.*)$" -AllMatches).Matches[0].Groups[1].Value
$elasticPassword | Out-File -FilePath "$es\config\elastic.password" -Encoding utf8
$kibanaPassword | Out-File -FilePath "$kibana\config\kibana.password" -Encoding utf8
  1. Remove setting xpack.ssl.key_passphrase from Elasticsearch.yml

  2. Restart Elasticsearch

  3. Verify Elasticsearch is work (and it is)

  4. Install X-Pack in kibana

&"$kibana\bin\kibana-plugin.bat" install x-pack

  1. Update Kibana.yml to below
server.name: devws-kibana
server.host: devws-kibana.local
elasticsearch.url: https://node1.local:9210/
elasticsearch.username: kibana
elasticsearch.password: nWD0zPDLFiM3yHdVQM9j
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt
  1. Start Kibana

  2. Verify Kibana is running and I am able to log in with elastic user

  3. Stop Kibana

  4. Update Kibana.yml to below

server.name: devws-kibana
server.host: devws-kibana.local
server.ssl.enabled: true
server.ssl.certificate: ../config/certs/devws-kibana.key
server.ssl.key: ../config/certs/devws-kibana.crt
elasticsearch.url: https://node1.local:9210/
elasticsearch.username: kibana
elasticsearch.password: nWD0zPDLFiM3yHdVQM9j
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt
xpack.security.encryptionKey: 3qrb1xee9ue9rrh3p93ykj28otgp676iu0l8ziifjopfov6h4sv9jhyp49gpm90t
  1. Try starting kibana. It fails and produces the following error.
FATAL Error: error:0906D06C:PEM routines:PEM_read_bio:no start line
    at Error (native)
    at Object.createSecureContext (_tls_common.js:69:17)
    at Server (_tls_wrap.js:776:25)
    at new Server (https.js:26:14)
    at Object.exports.createServer (https.js:47:10)
    at new module.exports.internals.Connection (C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\node_modules\hapi\lib\connection.js:88:74)
    at internals.Server.connection (C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\node_modules\hapi\lib\server.js:142:24)
    at KbnServer.exports.default (C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\http\setup_connection.js:43:10)
    at C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\kbn_server.js:171:20
    at next (native)
    at step (C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\kbn_server.js:87:191)
    at C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\kbn_server.js:87:437
    at C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\kbn_server.js:87:99
    at KbnServer.mixin (C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\kbn_server.js:187:7)
    at KbnServer.<anonymous> (C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\src\server\http\index.js:66:21)
    at next (native)

(Matt M) #2

Hit max body limit of 7000 characters. So here is some additional information that I wanted to include as I figure it is going to come up.

Documentation:


https://www.elastic.co/guide/en/kibana/6.1/installing-xpack-kb.html
https://www.elastic.co/guide/en/kibana/6.1/security-settings-kb.html
https://www.elastic.co/guide/en/kibana/current/production.html#enabling-ssl
https://www.elastic.co/guide/en/kibana/6.1/using-kibana-with-security.html
https://www.elastic.co/guide/en/kibana/6.1/settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/certutil.html

devws-kibana.key

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,17D5A5CA90BECF38

zOhDSJ8OFqB+FGvSmL18k4kIiojvO4yLnAmTYDar51NGAdvitBlGaOEvXzpBAx9j
3nwNVBvz+NCBMg9tOcr8UY9H1/Qgns7gpiWhCJmJlja33Jly97UJS9go6DHvWaJi
OoppKOHOkFaDDnMo6XkyHN2PXagVxFJxO8zW1OgQjMn1dWzUN8jwff/hUmgLT1tx
Arsso/+OKWEwZIJ2rczJjxXjgQmAu+d3libn8tLG9TpDStnj47j/8nxyViIp8Rq0
fFv3XW2QN8oj0Pjo0IBP8FjjGc8+b+zq+kgfvs0Lz65k5GigUcL66nkAgXTADb0S
1Qxo8NVgC4yzc94WrMqSiV99Tlw1kCh1Sb86TJDjYvWMko04wP+kFJrIPs2IxA9D
qxf7YwSZUHfIxg9FdNJHa+jHy+HinX9wOLK6JxZv5jWcZDNThjUW9JxRdvuM1nxD
6K0bUWAa8VlO0MU9x8Hwo7hhT4lJp6sr4jagKkvEqo/ZJWqyQzv5sOMQKu8vXGco
nqjxApjTQGIplf+RrYaPg7URH6GdOW3cIY0d+Hh9i1K/v7UolO2fZjSoCaYmt4YF
BjsMTIpdMnw1GfT0TlRa3bTgRAIqI9lE008fS/XRhjHgKY6wgHbbb7L+mxG05FIa
8rOAClu1btBJXZSELK4JqUgbY+kIPQqB2Ezh1FpggjTboJkdF2cXCazCLDtMSEi3
z95miNGm3rRtS0HNaqrElDQeNbbyo6X9joFKApaxvdICDe9rfdMbhl/tQkYZdyxA
XKeqbPL1dS95ma60A/7B0fIqwrD8Rko8q0EIMClG7bC/C7JRSdd8YEpIV9B1xorc
gYyPiLtoGsihFpF2wpapG0e45wKedtl5QAYbeMWp9JoRPx9ZUuaP4Usn6zdyCqtw
mWUGAHw2F7qQYpdvvUYrKAwC4dVRwivaHz4OoabppY0dYCq1THGvFBmlinbtDFLD
ihKwgnvdACyhWU+e45a53cAtpSwoaPVenFvUJU6MHt+XmxT757blb4Dn9nsHqrwz
HyWo0QoXdgbCwAzzBRW90R6wwYJx2cD+aYxpFop7aeeM/BgQ7IK2ejnykaTk3+NO
hTJ1RgUDvn30jmczvJE0hquzn4SxKVi/hn+YuPxENc2a3CDbj6vsLQRGYMxjnSwT
4POYxc+hgszaE7afh2sKjQHwNboHpeLNuFOXDU22StsIbltWltw+jKzz3sHNE83G
uUJnJPi7LN47WOT2pTNqV+k7/SEFfdo4JVDna54LF8LkrNLl9odmXR3ryTuTdQwP
fAiNXWySIKfEJufVb4g0tv3xDFtJztwZrVvOgCnMnlXw+4BWp7FmN3M+pjDReEdv
1t15tPoDltCfPRva4bZaUXriYSYK5w6PW0t33AR7WbUSKmYlisqInCXwACjBjQJc
nN8epVm33Kaqs18T1PSfkUKiqCKfFRqgleL5W96lG/W8b6qZvglod1qIu3k7qqiy
V6d/jLpe/lASCy1mCtjnNMYTkIlNc5Jyy7FrplXGEm5dWLunMCnCAYX4rfAunynW
1YdB/poa9pXBbUPrF1KxHVNAmV7x5VS4/0eK7LKcUAWs01nH7yiBsXEfs3YfsVMQ
-----END RSA PRIVATE KEY-----

devws-kibana.crt

-----BEGIN CERTIFICATE-----
MIIDbDCCAlSgAwIBAgIUMtq8dgtI+Opfy++z6Ptx8ci2Pr8wDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAxMTRWxhc3RpY3NlYXJjaC1EZXZXUzAeFw0xODAxMzAxOTUz
NTNaFw0yMTAxMjkxOTUzNTNaMBcxFTATBgNVBAMTDGRldndzLWtpYmFuYTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKxUb4Mc9PVov9+r6a7V1At1+JL/
CXl81g3tFQNYLzAMj/RJr06HAht8Loh7i7rBvWjleklOZNoKC8LMhbDNU56ix4WK
46yboRknYbjyEYOTjujx6gKoZ72yHugcbF5mc6GoafnyDJf0zL5nNHD8rC61WsFg
hUzJF0qlA9dMojjMMrlrDJDIeKRzhrDj950QBgC9bzLVHQ0sseHmmPQFRQRXXYiP
7/89jfrclgsRaB/H7A2DI5JnZz/HvTQ7UiVOoRXb4TW3NMc1OQbkTg2bjpPMruL6
tVXE/F0xLL4XXSUKbzT95p3vGxRp0jbnce0c8u4USiBEWoJbmYmD/rOouOECAwEA
AaOBqDCBpTAdBgNVHQ4EFgQUpbjtvSNDqoOha32Oculkz4L9c6kwWgYDVR0jBFMw
UYAU3G6UbG/SC9ucNlVGjGZmwk/CQaehIqQgMB4xHDAaBgNVBAMTE0VsYXN0aWNz
ZWFyY2gtRGV2V1OCFQDW6SMoh4MojBt5XyEkdFAA85u/jTAdBgNVHREEFjAUghJk
ZXZ3cy1raWJhbmEubG9jYWwwCQYDVR0TBAIwADANBgkqhkiG9w0BAQsFAAOCAQEA
Nr0wX2LqbIV8c4/tXaPUA34J8WoWS0HarADrFA+NFee3+/V0Afoun20HI8B8ql6a
5sK+omjDTsHxjjGIitatS/ntmOacJST4MLVeQzSHEfx3ArvozD0LJqnpCifvvUMz
0Gq1fmZHUioz3sHeqw4ZlpEll8MRyvtVj6gDQqzUJcaw9S06/J3VsEJbrRudhSLD
GYgCz1P5jIsZoiEFhinWafgayImcYY3LMNdpRFwIbmwO4Uhs26vgt2zfRVW3vw6r
+5O+DMvjoSe46fZN4uY36rCAEiCoYDC+PB6LSFOGbGswylTTL6F8EorQtjJxOnsn
yFmugwITE4K4iwAQcuCy+g==
-----END CERTIFICATE-----

ca.crt

-----BEGIN CERTIFICATE-----
MIIDWzCCAkOgAwIBAgIVANbpIyiHgyiMG3lfISR0UADzm7+NMA0GCSqGSIb3DQEB
CwUAMB4xHDAaBgNVBAMTE0VsYXN0aWNzZWFyY2gtRGV2V1MwHhcNMTgwMTMwMTk1
MzQ5WhcNMjEwMTI5MTk1MzQ5WjAeMRwwGgYDVQQDExNFbGFzdGljc2VhcmNoLURl
dldTMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArB0kt1hdL0xuCqtw
pO/9wr6xvL9uOy63uHWhEvF8hqW1Kv3w9unZQkJxXlQj3oVbhJTGD6+Bu6RZ8nwl
J7kVjf3EHDvcvwZZElHjmV0zlZ8k3XlJJmKIFeCNAa52YpmReiFerIv+xWV9F4Ae
B77O2pNzfvrJzWroPBVodbF9/N0kxplwSbAJPRGLDvknxW0vX3XiyjvDUPZkmVhm
xc7g0XkTqtjGcYKylz3sfCEnBOSY+3TKePyA62thKlmfMb5iDxGHjraHCcXzPtjh
y3LcD4E3KM57xv1XnHyrKxzJLf0iaJb1xyd4aRGFfckhkqrGvyaS08PRLd3RL+QE
/JsRKQIDAQABo4GPMIGMMB0GA1UdDgQWBBTcbpRsb9IL25w2VUaMZmbCT8JBpzBa
BgNVHSMEUzBRgBTcbpRsb9IL25w2VUaMZmbCT8JBp6EipCAwHjEcMBoGA1UEAxMT
RWxhc3RpY3NlYXJjaC1EZXZXU4IVANbpIyiHgyiMG3lfISR0UADzm7+NMA8GA1Ud
EwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAIA7S3HbOEKw9kTjxPYlOYoe
kQiKTyZ7rxUAJYSnafnmsjbCMbpVXS9k+THm3IUTQuyxwrGuuBBeKFZJWQ1FcPiF
DVPsgsNO4MRT6r78XjmCJJZcB5FZqbfK7EQd/E4sRzq8bk5VR6wfQK/U5/42TTcw
5RdDYnS4axLQOb9AuSdma7XP6BcshNAFCTp39caP7ZfKLJeRMMv0Mn0/3Yt9I9dv
2MGpxnMOYeVKzYVeoyXXDIOZqdPEkPO6gO7i1MprHcC3XlXFwkbe/EZ4pKUtRTJU
kUgoSTOEd8BO8hwOYhG3HjOqTQe4U6lp2J58Kk47MMs8KUH5Zv47O8baNdHPWVw=
-----END CERTIFICATE-----

(Brandon Kobel) #3

Hey @m_ebags, can you try using an absolute filepath for the following settings in your kibana.yml:

server.ssl.certificate: ../config/certs/devws-kibana.key
server.ssl.key: ../config/certs/devws-kibana.crt
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt

(Matt M) #4

As a first step I changed the cert/key paths to absolute paths and added settings logging.verbose and logging.dest to my kibana.yml file but left out the server.ssl.* settings to verify the logging and absolute paths were working as expected.

logging.verbose: true
logging.dest: C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\logs\kibana-log.txt
server.name: devws-kibana
server.host: devws-kibana.local
elasticsearch.url: https://node1.local:9210/
elasticsearch.username: kibana
elasticsearch.password: nWD0zPDLFiM3yHdVQM9j
elasticsearch.ssl.certificateAuthorities: C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\config\certs\ca.crt
xpack.security.encryptionKey: 3qrb1xee9ue9rrh3p93ykj28otgp676iu0l8ziifjopfov6h4sv9jhyp49gpm90t

Next I added the server.ssl settings and ran kiban again. However this time kibana failed to create a log file and instead wrote the "FATAL Error: error:0906D06C:PEM routines:PEM_read_bio:no start line" error to the console. It appears kibana failed before it had a chance to write anything to the log file.

logging.verbose: true
logging.dest: C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\logs\kibana-log.txt
server.name: devws-kibana
server.host: devws-kibana.local
server.ssl.enabled: true
server.ssl.certificate: C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\config\certs\devws-kibana.key
server.ssl.key: C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\config\certs\devws-kibana.crt
elasticsearch.url: https://node1.local:9210/
elasticsearch.username: kibana
elasticsearch.password: nWD0zPDLFiM3yHdVQM9j
elasticsearch.ssl.certificateAuthorities: C:\working\elasticsearch\kibana-6.1.1-windows-x86_64\config\certs\ca.crt
xpack.security.encryptionKey: 3qrb1xee9ue9rrh3p93ykj28otgp676iu0l8ziifjopfov6h4sv9jhyp49gpm90t

(Matt M) #5

Figured out the issue. the server.ssl.certificate and server.ssl.key values were switched.


(Jared Carey) #6

Please make sure to discard the private key (devws-kibana.key) / public cert (devws-kibana.key) that you provided here - since that private key is now public. :slight_smile:


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.