Hi,
We encountered an issue after switch to the trial license from the basic license, the ssl connection of the http layer don't work anymore from remote Kibana and Logstash nodes.
I didn't see any problem for the transport layer, synchronization between elasticsearch nodes was OK.
## Request from one of the Elasticsearch node to one of the Elasticsearch node
No problem, 0 error log.
Trial license has been actived :
GET https://elastic-node1:9200/_xpack/license
{
"license" : {
"status" : "active",
"uid" : "*********************",
"type" : "trial",
"issue_date" : "2019-12-04T09:43:49.280Z",
"issue_date_in_millis" : 1575452629280,
"expiry_date" : "2020-01-03T09:43:49.280Z",
"expiry_date_in_millis" : 1578044629280,
"max_nodes" : 1000,
"issued_to" : "elastic-cluster",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
GET https://elastic-node1:9200/_cluster/health?pretty -v
* Trying 10.3.144.143...
* TCP_NODELAY set
* Connected to elastic-node1 (10.3.144.143) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=***; ST=***; L=***; O=***; OU=***; CN=elastic-node1
* start date: Sep 25 05:46:03 2019 GMT
* expire date: Sep 24 05:46:03 2021 GMT
* subjectAltName: host "elastic-node1" matched cert's "elastic-node1"
* issuer: C=***; ST=***; L=***; O=***; OU=***; CN=***; emailAddress=***
* SSL certificate verify ok.
* Server auth using Basic with user 'kibana'
> GET /_cluster/health?pretty HTTP/1.1
Host: elastic-node1:9200
> Authorization: Basic a2liYWk12kQ0W1JmQ2N5QS1qNmVZUEctStpx
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 489
<
{
"cluster_name" : "elastic-cluster",
"status" : "green",
"timed_out" : false,
...
}
* Curl_http_done: called premature == 0
* Connection #0 to host elastic-node1 left intact
## Request from Kibana or Logstash node
GET https://elastic-node1:9200/_cluster/health?pretty -v
> * Trying 10.3.144.145...
> * TCP_NODELAY set
> * Connected to elastic-node1 (10.3.144.145) port 9200 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> * CAfile: /etc/ssl/certs/ca-certificates.crt
> CApath: /etc/ssl/certs
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * Unknown SSL protocol error in connection to elastic-node1:9200
> * Curl_http_done: called premature == 0
> * Closing connection 0
> curl: (35) Unknown SSL protocol error in connection to elastic-node1:9200
openssl s_client -connect elastic-node1:9200
> CONNECTED(00000003)
> write:errno=0
> ---
> no peer certificate available
> ---
> No client certificate CA names sent
> ---
> SSL handshake has read 0 bytes and written 176 bytes
> Verification: OK
> ---
> New, (NONE), Cipher is (NONE)
> Secure Renegotiation IS NOT supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : 0000
> Session-ID:
> Session-ID-ctx:
> Master-Key:
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1575537612
> Timeout : 7200 (sec)
> Verify return code: 0 (ok)
> Extended master secret: no
> ---
Log kibana-node2 :
{"type":"log","@timestamp":"2019-12-04T09:52:00Z","tags":["warning","elasticsearch","security"],"pid":18465,"message":"Unable to revive connection: https://elastic-node1:9200/"}
{"type":"log","@timestamp":"2019-12-04T09:52:00Z","tags":["warning","elasticsearch","security"],"pid":18465,"message":"Unable to revive connection: https://elastic-node2:9200/"}
{"type":"log","@timestamp":"2019-12-04T09:52:00Z","tags":["warning","elasticsearch","security"],"pid":18465,"message":"Unable to revive connection: https://elastic-node3:9200/"}
{"type":"log","@timestamp":"2019-12-04T09:52:00Z","tags":["warning","elasticsearch","security"],"pid":18465,"message":"Unable to revive connection: https://elastic-node4:9200/"}
{"type":"log","@timestamp":"2019-12-04T09:52:00Z","tags":["warning","elasticsearch","security"],"pid":18465,"message":"No living connections"}
{"type":"log","@timestamp":"2019-12-04T09:52:03Z","tags":["error","elasticsearch","data"],"pid":18465,"message":"Request error, retrying\nGET https://elastic-node1:9200/_xpack => Client networ
k socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2019-12-04T09:52:03Z","tags":["error","elasticsearch","data"],"pid":18465,"message":"Request error, retrying\nGET https://elastic-node1:9200/_xpack => Client networ
k socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2019-12-04T09:52:03Z","tags":
Kibana Portal is not ready...
Logstash worked well until elastic-node1 reboot bc connection has been removed. Connection can't be restablished anymore with elastic-node1.
Log logstash-node1 :
{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1575471499421,"thread":"Ruby-0-Thread-56: :1","logEvent":{"message":"Attempted to resurrect connection to dead ES instance, but got an error.","url":"https://logstash_int:xxxxxx@elastic-node1:9200/","error_type":{"metaClass":{"metaClass":{"error_type":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError","error":"Elasticsearch Unreachable: [https://logstash_int:xxxxxx@elastic-node1:9200/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}}}}}
Log Kibana-node2 when I activated the trial license :
{"type":"response","@timestamp":"2019-12-04T09:43:31Z","tags":[],"pid":18465,"method":"get","statusCode":200,"req":{"url":"/api/license/start_trial?_=1575452598602","method":"get","headers":{"host":"kibana-node2:5601","connection":"keep-alive","accept":"*/*","kbn-version":"7.4.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","content-type":"application/json","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://kibana-node2:5601/s/optilian/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7,pt;q=0.6"},"remoteAddress":"192.168.50.5","userAgent":"192.168.50.5","referer":"https://kibana-node2:5601/s/optilian/app/kibana"},"res":{"statusCode":200,"responseTime":87,"contentLength":9},"message":"GET /api/license/start_trial?_=1575452598602 200 87ms - 9.0B"}
{"type":"response","@timestamp":"2019-12-04T09:43:38Z","tags":[],"pid":18465,"method":"get","statusCode":200,"req":{"url":"/built_assets/dlls/icon.cross-js.bundle.dll.js","method":"get","headers":{"host":"kibana-node2:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","referer":"https://kibana-node2:5601/s/optilian/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7,pt;q=0.6"},"remoteAddress":"192.168.50.5","userAgent":"192.168.50.5","referer":"https://kibana-node2:5601/s/optilian/app/kibana"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /built_assets/dlls/icon.cross-js.bundle.dll.js 200 4ms - 9.0B"}
{"type":"log","@timestamp":"2019-12-04T09:43:49Z","tags":["error","elasticsearch","data"],"pid":18465,"message":"Request error, retrying\nGET https://elastic-node2:9200/_xpack => Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2019-12-04T09:43:49Z","tags":["error","elasticsearch","data"],"pid":18465,"message":"Request error, retrying\nGET https://elastic-node1:9200/_xpack => Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2019-12-04T09:43:49Z","tags":["license","info","xpack"],"pid":18465,"message":"Imported changed license information from Elasticsearch for the [data] cluster: mode: trial | status: active | expiry date: 2020-01-03T10:43:49+01:00"}
{"type":"log","@timestamp":"2019-12-04T09:43:49Z","tags":["info","monitoring","kibana-monitoring"],"pid":18465,"message":"Starting monitoring stats collection"}
{"type":"response","@timestamp":"2019-12-04T09:43:49Z","tags":[],"pid":18465,"method":"post","statusCode":200,"req":{"url":"/api/license/start_trial","method":"post","headers":{"host":"kibana-node2:5601","connection":"keep-alive","content-length":"0","accept":"*/*","origin":"https://kibana-node2:5601","kbn-version":"7.4.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","content-type":"application/json","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://kibana-node2:5601/s/optilian/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7,pt;q=0.6"},"remoteAddress":"192.168.50.5","userAgent":"192.168.50.5","referer":"https://kibana-node2:5601/s/optilian/app/kibana"},"res":{"statusCode":200,"responseTime":484,"contentLength":9},"message":"POST /api/license/start_trial 200 484ms - 9.0B"}
{"type":"log","@timestamp":"2019-12-04T09:43:49Z","tags":["error","elasticsearch","security"],"pid":18465,"message":"Request error, retrying\nGET https://elastic-node2:9200/_security/privilege/kibana-.kibana => Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2019-12-04T09:43:49Z","tags":["error","elasticsearch","security"],"pid":18465,"message":"Request error, retrying\nGET https://elastic-node1:9200/_security/privilege/kibana-.kibana => Client network socket disconnected before secure TLS connection was established"}
## Rollback Basic license, SSL OK
After removing the trial license, kibana respond again but notify that elasticsearch dont have information license.
DELETE https://elastic-node1:9200/_license
GET https://elastic-node1:9200/_license
{}
Log Kibana :
{"type":"log","@timestamp":"2019-12-05T11:01:25Z","tags":["status","plugin:security@7.4.2","error"],"pid":23405,"state":"red","message":"Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Log Logstash
{level":"WARN","loggerName":"logstash.licensechecker.xpackinfo","timeMillis":1575543990130,"thread":"monitoring-license-manager","logEvent":{"message":"Nil response from License Server"}}
After generating new basic license, communications are working again.
POST https://elastic-node1:9200/_license/start_basic
{"acknowledged":true,"basic_was_started":true}
GET https://elastic-node1:9200/_xpack/license
{
"license" : {
"status" : "active",
"uid" : "***************",
"type" : "basic",
"issue_date" : "2019-12-05T11:07:55.561Z",
"issue_date_in_millis" : 1575544075561,
"max_nodes" : 1000,
"issued_to" : "elastic-cluster",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
I tried to upgrade elasticsearch-node to 7.5 before removing the trial license, same result.
Jeremy.