Kibana shows elasticsearch unavailable after SSL enabled

[KeninHouston (Kenneth)]

Hi all,

I have successfully configured SSL encryption for my elasticsearch cluster, but when I change the kibana.yml file to connect to the elasticsearch cluster, kibana starts, but it shows elasticsearch as service unavailable.

My elasticsearch version is 2.4 and the kibana version is 4.6.1

The contents of my kibana.yml are:

elasticsearch.url: "https://10.4.0.4:9200"
logging.dest: /var/log/kibana.log
elasticsearch.username: es_kibana_server
elasticsearch.password:
shield.useUnsafeSessions: true
shield.skipSslCheck: true
shield.encryptionKey: "noozaibesohzohduolohgheedooleejietaeruokooyeesohphaixaipeituthuu"
reporting.encryptionKey: "johxaecaiheelaipoagawaiquawaishonienipiezootahlooxueghaiqueerohc"
sense.defaultServerUrl: "https://10.4.0.4:9200"
server.ssl.key: /opt/kibana/config/owaUSkibana.key
server.ssl.cert: /opt/kibana/config/owaUSkibana.crt
elasticsearch.ssl.ca: /opt/kibana/config/ca.crt

The kibana log shows nothing I'm afraid.

Kibana works just fine when I disable the SSL communication in elasticsearch however, so it's definitely a problem communicating with the elasticsearch cluster.

Any ideas?

[LeeDr (Lee Drengenberg)]

Hi Kenneth,

What OS are you on?

Can you show us the relevant parts of your Elasticsearch.yml also (the ssl parameters).

Is everything good in the elasticsearch.log? Any messages about ssl parameters or certificates?
We might need to increase the log level there.

Same with kibana.stdout. There must be something in it about failing to connect to elasticsearch? Can you paste that here?

From the Kibana server, can you connect to Elasticsearch at https://10.4.0.4:9200 with that es_kibana_server user and password?

Regards,
Lee

[KeninHouston (Kenneth)]

Hi Lee,

The operating system is Ubunto 14. Here's the elasticsearch.yml entries

cluster.name: owaNAM
node.name: owaUSdata-0
path.data: /datadisks/disk1/elasticsearch/data
discovery.zen.ping.unicast.hosts: ["owaUSdata-0:9300","owaUSdata-1:9300","owaUSdata-2:9300"]
node.master: true
node.data: true
index.number_of_shards: 5
index.number_of_replicas: 1
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
network.host: non_loopback
marvel.agent.enabled: true
node.max_local_storage_nodes: 1
shield.ssl.keystore.path: /etc/elasticsearch/config/truststore.jks
shield.ssl.keystore.password: owastore
shield.transport.ssl: true
shield.http.ssl: true
shield.http.ssl.client.auth: optional
shield.authc.pki.files.role_mapping: /etc/elasticsearch/shield/role_mapping.yml
shield:
authc:
realms:
pki1:
type: pki
order: 0
file1:
type: file
order: 1
native1:
type: native
order: 2

I've looked and I can't see any logs created at all by the elasticsearch server. I found the logging.yml file which seems to have the default values.

you can override this using by setting a system property, for example -Des.logger.level=DEBUG

es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

deprecation logging, turn to DEBUG to see them

deprecation: INFO, deprecation_log_file

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

aws will try to do some sketchy JMX stuff, but its not needed.

com.amazonaws.jmx.SdkMBeanRegistrySupport: ERROR
com.amazonaws.metrics.AwsSdkMetrics: ERROR

org.apache.http: INFO

gateway

#gateway: DEBUG
#index.gateway: DEBUG

peer shard recovery

#indices.recovery: DEBUG

discovery

#discovery: TRACE

index.search.slowlog: TRACE, index_search_slow_log_file
index.indexing.slowlog: TRACE, index_indexing_slow_log_file

additivity:
index.search.slowlog: false
index.indexing.slowlog: false
deprecation: false

I've checked and the kibana server log is empty as well. Not sure why this is? I've tried killing the kibana process and it automatically starts up again.

I can connect from the kibana server to the elasticsearch successfully

owa_admin@owaUSkibana:~$ curl --cacert ca.crt -XGET -u es_kibana_server https://owaUSdata-0:9200/_nodes?pretty
Enter host password for user 'es_kibana_server':
{
"cluster_name" : "owaNAM",
"nodes" : {
"Wcyzdp0JSF2fFvq3M_VRpQ" : {
"name" : "owaUSdata-0",
"transport_address" : "10.4.0.8:9300",
"host" : "10.4.0.8",
"ip" : "10.4.0.8",
"version" : "2.4.0",
"build" : "ce9f0c7",
"http_address" : "10.4.0.8:9200",
"attributes" : {
"max_local_storage_nodes" : "1",
"master" : "true"
},
"settings" : {
"cluster" : {
"name" : "owaNAM"
},
"shield" : {
"http" : {
"ssl" : "true",
"client" : {
"auth" : "optional"

Sorry, but I'm very new to elasticsearch

Regards,
Kenneth