Kibana can't reach Elasticsearch cluster with security enabled

[BenP]

Hi,

Since I reverted from my trial license to a basic one, Kibana can't reach my Elasticsearch cluster as long as "xpack.security.enabled: true". From what I understood, I must configure "at least" TLS between my nodes and I followed the instructions here : Encrypting communications in Elasticsearch but I still have the same error.
I'm probably missing something simple but I can't find it.

kibana.log :

{"type":"log","@timestamp":"2019-09-10T07:46:43Z","tags":["warning","elasticsearch","admin"],"pid":1275,"message":"No living connections"}
{"type":"log","@timestamp":"2019-09-10T07:46:43Z","tags":["warning","elasticsearch","admin"],"pid":1275,"message":"Unable to revive connection: http://10.1.0.4:9200/"}

kibana.yml :

elasticsearch.hosts: "http://10.1.0.4:9200"
server.host: 10.1.0.7
logging.dest: /var/log/kibana.log
logging.quiet: false
elasticsearch.username: "kibana"
elasticsearch.password: "mypassword"
xpack.security.encryptionKey: "longkey"
xpack.reporting.encryptionKey: "longkey"

elasticsearch.hosts point to my internal load balancer.
Do I have to use TLS between Kibana and my nodes too ?

elasticsearch.yml :

cluster.name: "elastic-dev"
node.name: "devdata-0"
path.logs: /var/log/elasticsearch
path.data: /datadisks/disk1/elasticsearch/data
discovery.zen.ping.unicast.hosts: ["devdata-0:9300","devdata-1:9300","devdata-2:9300"]
node.master: true
node.data: true
discovery.zen.minimum_master_nodes: 2
#network.host: [site, local]
network.host: 0.0.0.0
node.max_local_storage_nodes: 1
node.attr.fault_domain: 0
node.attr.update_domain: 0
cluster.routing.allocation.awareness.attributes: fault_domain,update_domain
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: ${node.name}.p12
xpack.security.transport.ssl.truststore.path: ${node.name}.p12
bootstrap.memory_lock: true

Everything is hosted on Azure and have been deployed thanks to the deployment template on Azure Marketplace.

Thanks in advance.

[ikakavas (Ioannis Kakavas)]

Is Elasticsearch running at all ? If it is please check the logs, I would assume you will find a few errors in there that would point you to the actual issue at hand.

[BenP]

Look like you were right, I had a permission issue on my .p12 files preventing Elasticsearch to start.

I resolved it and now I it looks like I have messed up the password I entered when I ran :
bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

Since I'm getting in elasticsearch logs :

java.io.IOException: keystore password was incorrect

[BenP]

I solved my certificate problem but now Kibana is throwing an error when I'm logging in with the kibana user :
{"statusCode":403,"error":"Forbidden","message":"Forbidden"}

The kibana user permissions haven't changed so I don't get why it would throw a "Forbidden".

EDIT: I just tried to login with the elastic user and it works.

[ikakavas (Ioannis Kakavas)]

Just so that this is clear for others reading the forums, you are not supposed to log in with the kibana user. This is the internal user that Kibana server uses to communicate with Elasticsearch.

One should log in with the elastic user first and then create all the necessary users they would need to operate/administer/use the cluster and log in with those from then on.

[BenP]

Well, thanks a lot for your time, my problem is solved.