Elasticsearch not working after enabling X-Pack

Hello,
I have installed elasticsearch version 6.8.0 in my production and have created a cluster with 3 master nodes and 20 data nodes. I have installed x-pack in all of the nodes and was trying to authenticate the cluster. For that, I enabled xpack security on 1 master node in the elasticsearch.yml file.
For making changes to the config file and authenticating the node, I followed the below documentation:

After following those steps, when I ran the following command:
sudo bin/elasticsearch-setup-passwords interactive

I received the following error:
ERROR: Failed to connect to elasticsearch at http://127.0.0.1:9200/_xpack/security/_authenticate?pretty. Is the URL correct and elasticsearch running?

For resolving this, I followed the below documentation:
https://www.elastic.co/guide/en/elastic-stack-overview/6.8/trb-security-setup.html

but could not solve the problem. Upon investigating further, I found that I don't have enough permissions to access the
"elasticsearch.keystore" file. For that, I tried running the two commands:

sudo chown app:app /usr/local/etc/elasticsearch

sudo chmod g+ws /usr/local/etc/elasticsearch

" /usr/local/etc/elasticsearch" is the folder containg the "elasticsearch.keystore" file.
But still, I am not able to resolve the issue.

Can anyone please help me?

Thanks.
Dhrumil

The first place to start is to answer those questions.

  • Do you expect your node to be accessible via http://127.0.0.1:9200 ? (Is it really bound to the local interface? Is it on port 9200?)
  • If so, is your node running? What do the logs say?

@TimV Thanks for the reply.
I have taken care of the above issue. But, moving forward I am currently getting the following error when I am trying to run "elasticsearch-setup-passwords" command:

" SSL connection to https://10.12.218.37:9200/_xpack/security/_authenticate?pretty failed: java.security.cert.CertificateException: No subject alternative names present

Please check the elasticsearch SSL settings under xpack.security.http.ssl.

ERROR: Failed to establish SSL connection to elasticsearch at https://10.12.218.37:9200/_xpack/security/_authenticate?pretty. "

In order to resolve this error, I followed this documentation and especially followed point 3:
https://www.elastic.co/guide/en/elastic-stack-overview/master/trb-security-setup.html

But still, I am unable to resolve the errors.
Is there an issue with the certificate I have installed?

For installing the certificate, I have followed the steps described here:

@TimV I created those certificates again with the "--ip" parameter and then tried again setting up the password through "elastic-setup-passwords".
But, this time I received the following error:
"SSL connection to https://10.12.218.37:9200/_xpack/security/_authenticate?pretty failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Please check the elasticsearch SSL settings under xpack.security.http.ssl. "

I have already set those parameters in config file.
Currently, I am able to open the node in the browser with the link: https://10.12.218.37:9200

But, for all the username and password combination it is failing and I am unable to access the cluster..

I tried setting up the "elastic" username's password by setting up the bootstrap password:
sudo bin/elasticsearch-keystore add "bootstrap.password"

I also tried adding few more superusers with the following command:
sudo bin/elasticsearch-users useradd user -p password -r superuser

Those users are added properly as I am able to confirm it with then following command:
sudo bin/elasticsearch-users list

Still, I am unable to access my clusters as adding those username and passwords is not giving me access to the cluster.

Below is the config file:
'''
cluster.name: elasticsearch
node.name: udl47Q8
node.master: true
node.data: true
node.attr.rack: dfw
cluster.routing.allocation.awareness.force.cloud.values: rack
path.data: /data/elasticsearch
path.logs: /data/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.12.218.37
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.12.11.60", "10.12.15.110", "10.12.137.203", "10.12.11.89", "10.12.11.97", "10.12.120.219", "10.12.11.57", "10.12.15.92", "10.12.14.20", "10.12.15.178", "10.12.15.179", "10.12.30.211", "10.12.30.209", "10.12.30.210", "10.12.30.201","10.12.11.118", "10.12.13.103", "10.12.30.202", "10.12.24.223"]
#discovery.zen.minimum_master_nodes: 2
#gateway.recover_after_nodes: 3
#gateway.expected_nodes: 1
action.destructive_requires_name: true
http.cors.allow-origin: "*"
http.cors.enabled: true

#settings for x-pack
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/local/etc/elasticsearch/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: /usr/local/etc/elasticsearch/elastic-stack-ca.p12
xpack.security.http.ssl.enabled : true
xpack.security.http.ssl.verification_mode: full
xpack.security.http.ssl.keystore.path: /usr/local/etc/elasticsearch/elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: /usr/local/etc/elasticsearch/elastic-stack-ca.p12
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
'''

In my logs, I am also receiving the following error:
" Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record elasticsearch "

Which parameters did you set?
Please be specific in your posts - we cannot give you good advice if we're guessing.

These all look to be incorrect. You should not be using a CA keystore for any of these settings.
The blog post you claim to be following does not do that.