Newbie to Linux/elastic. I'm having trouble setting up elasticsearch

As the title says I'm new to Elasticsearch and Linux in general. I have setup an Ubuntu server in VirtualBox for installing Elasticsearch. I followed the guide listed on installing it and am currently stuck on this step.

This is the command I ran. Only thing I changed was the IP.

root@scrub-ubuntu-server:/etc/elasticsearch# curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic https://10.0.2.15:9200
Enter host password for user 'elastic':
curl: (77) error setting certificate verify locations:
  CAfile: /etc/elasticsearch/config/certs/http_ca.crt
  CApath: /etc/ssl/certs

Can anyone offer guidance on how to get past that error?

Welcome to our community! :smiley:

Does it give you the error if you don't pass in the --cacert?

I tried without --cacert first and got the below message:

root@scrub-ubuntu-server:~# curl -u elastic https://10.0.2.15:9200
Enter host password for user 'elastic':
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Then I tried the following without https:

root@scrub-ubuntu-server:~# curl -u elastic http://10.0.2.15:9200
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Aroro

Tried without specifying "http." (Not sure really sure of the difference between having http and just having the IP if someone is able to point out the difference).

root@scrub-ubuntu-server:~# curl -u elastic 10.0.2.15:9200
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401}

You need the password try it inline.

What version? 8.0?

Or other version did you specifically set up https?

Try http or https depending

curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic:<password> http://localhost:9200

It looks like that you don’t need to use https. Elasticsearch 8.0.0, for which you are using the documentation,is installed with security features enabled and TLS for the HTTP layer is enabled by default ( this is why you would need to pass the —cacert option ). But did you install version 8.0.0 or a previous version ?

1 Like

I installed the latest version and following along the documentation which lists the --cacert option.

root@scrub-ubuntu-server:/usr/share/elasticsearch# sudo ./bin/elasticsearch --version
Version: 8.0.1, Build: default/deb/801d9ccc7c2ee0f2cb121bbe22ab5af77a902372/2022-02-24T13:55:40.601285296Z, JVM: 17.0.2

Yes, version 8.0.1. I'm not sure if having the password inline matters since I still had to enter the password even if I didn't put it inline.

I tried both with http and https.

root@scrub-ubuntu-server:/usr/share/elasticsearch# curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic:redacted http://10.0.2.15:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401}
root@scrub-ubuntu-server:/usr/share/elasticsearch#
root@scrub-ubuntu-server:/usr/share/elasticsearch# curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic:redacted https://10.0.2.15:9200
curl: (77) error setting certificate verify locations:
  CAfile: /etc/elasticsearch/config/certs/http_ca.crt
  CApath: /etc/ssl/certs

So as you see, same errors as I was getting in the original post and 3rd reply

1st 8.0.1 Means https is enabled by default on Elasticsearch (not kibana) unless you did not do default install instructions

2nd /etc/elasticsearch/config/certs/http_ca.crt what are the permission does the current user have permissions to access the certs? seems like it does not per the error message

try

sudo curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic https://10.0.2.15:9200

Maybe let us take a step back here :slight_smile:

  • From the requests, we can clearly see that Elasticsearch replies when accessed over http. That means that TLS is not enabled. This means that either we didn't do the auto-configuration on startup or the user has disabled HTTPS after installation. The former would explain why curl fails too, it looks as if /etc/elasticsearch/config/certs/http_ca.crt doesn't exist.

  • They seem to be root already, so sudo wouldn't help.

@elasticScrub can you please share some more information and show us your configuration ?

  • How did you install Elasticsearch ( the actual commads you used ).
  • Did you get a message like this on installation ?
  • Was this an upgrade over an existing version? Did you maybe install a 7.x first and then 8.0.1 on top of it ?

What is the output of the following commands:

cat /etc/elasticsearch/elasticsearch.yml
ls -laR  /etc/elasticsearch/config

@ikakavas I can chime in as I'm also experiencing this repeatedly on fresh installs of Ubuntu server with 8.0.1.

commands executed:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install elasticsearch

At this point I did receive the Security autoconfiguration information" message as pasted below, but for the password:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : <password>

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Finally, I ran:

curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic https://localhost:9200 

Where I received this output:

Enter host password for user 'elastic':
curl: (77) error setting certificate verify locations:
  CAfile: /etc/elasticsearch/config/certs/http_ca.crt
  CApath: /etc/ssl/certs

I attempted:

sudo apt install ca-certificates --reinstall

but that did not help.

cat /etc/elasticsearch/elasticsearch.yml reveals:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#

# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 05-03-2022 04:54:44
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["elasticserver"]

# Allow HTTP API connections from localhost and local networks
# Connections are encrypted and require user authentication
http.host: [_local_, _site_]

# Allow other nodes to join the cluster from localhost and local networks
# Connections are encrypted and mutually authenticated
#transport.host: [_local_, _site_]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

which seems reasonable / as expected at this point.

ls -laR  /etc/elasticsearch/config

Results in:

ls: cannot access '/etc/elasticsearch/config': No such file or directory

which...... seems odd since we would expect it to be there

If I attempt to curl insecurely with curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic https://localhost:9200 -k
I get:

Enter host password for user 'elastic':
{
  "name" : "elasticserver",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "KZc0isHyQL24ZvxQ0ZEmkQ",
  "version" : {
    "number" : "8.0.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "801d9ccc7c2ee0f2cb121bbe22ab5af77a902372",
    "build_date" : "2022-02-24T13:55:40.601285296Z",
    "build_snapshot" : false,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

SO. It does seem like there's some certificate wonky-ness here.

I ran find . -name '*.p12' and get:

./etc/elasticsearch/certs/http.p12
./etc/elasticsearch/certs/transport.p12

Also in that same directory is http_ca.crt

I did a mkdir for /etc/Elasticsearch/config/certs/ and then ran cp http_ca.crt /etc/elasticsearch/config/certs/http_ca.crt and that seems to resolve the issue, but it almost seems like a hack, so I'm left wondering if things will still run as expected going forward, as this step was not mentioned in the install instructions.

I am unclear why you think that is a "Hack" you need to provide the path to the http_ca.crt

The instruction even say

curl --cacert /etc/Elasticsearch/config/certs/http_ca.crt -u elastic https://localhost:9200

Ensure that you use https in your call, or the request will fail.

--cacert

Path to the generated http_ca.crt certificate for the HTTP layer.

It is instructing you to copy or move the http_ca.crt to this path /etc/elasticsearch/config/certs/ which is typical location for it ... or you can leave it wherever you generated and then put the path in the curl to whereever it is ... it does not automatically move to that path.

Well, because it was midnight and I needed a break so I definitely didn't see that :sweat_smile: so apologies for that. I had definitely assumed that the curl command in the instructions would just point to where the .crt file was dropped and the lazy copy/paste approach would just work.

Thank you for your input.

As a potentially related aside, when trying to enroll and start the elastic agent on another host to check in with the fleet server, I'm currently stuck in a loop of "Remote server is not ready to accept connections, will retry in a moment" loop upon running the
sudo ./elastic-agent install --url=https://fleetserverip:443 --enrollment-token=NHUyX1dIOEJXZ0RsS3RDMDgtRlg6SEZGRTBEd2lSNjZRWWxGQl9HU05VQQ==
command. running a curl on the IP prints "Connection refused," and decoding that token doesn't seem to provide anything useful.

I read through the troubleshooting guide and the seemingly most appropriate thing to try was running ./elastic-agent install -f
I followed that with

elastic-agent enroll -f --fleet-server-es=https://'elasticsearchip':443 --fleet-server-service-token=<token> --fleet-server-policy="Default policy"

(I'm assuming that's what it's asking for with the policy, but definitely am not sure)

and first get a connection refused message, followed by a Error: fail to enroll: fail to execute request to fleet-server: context canceled message.

when I change the ip to the fleet server ip I get

"message":"Fleet Server - Stopping"
"message":"Fleet Server - Restarting"
Error: fleet-server failed: context canceled

Do I also need to copy a certificate for each machine that will have an agent? Any ideas? I'm happy to start a new thread if needed but thought it might potentially be related to this sort of thing

I would suggest opening a new thread with a good/ descriptive title you will probably get more eyes. Fleet is not my expertise.

Describe your current setup and the issue include the versions etc.

And BTW we have all been blurry eyed at midnight at the keyboard :slight_smile:

1 Like

No doubt. Typing up a new thread now, thank you!

@epheria , @elasticscrub it seems this is just a docs error in

curl --cacert /etc/elasticsearch/config/certs/http_ca.crt -u elastic https://localhost:9200 

should be

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200 

Apologies for missing this above. We will adjust the docs, thanks for bringing this up !

1 Like

@ikakavas

Sorry for the late reply, I didn't get a chance to work on this much and I tried a fresh install. I followed all the steps @epheria did until the curl command. And nice that the docs got adjusted!

I'm still getting an error that I got previously though. I followed the updated command as well.

root@ubuntu:~# curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://10.0.2.15:9200
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

The only changes I have done are in Elasticsearch.yml file.

  • I uncommented the network host line and changed it to 10.0.2.15.
  • I uncommented the http port.
  • In the Discovery section, I added discovery.type: single-node

Edit:
I rebooted the server and tried starting Elasticsearch.service and was greeted with this message in the logs when I ran journalcte -xe

java.lang.IllegalArgumentException: setting [cluster.initial_master_nodes] is not allowed when [discovery.type] is set to [single-node]

I don't get how this is coming up since I added discovery.type: single-node

Here is my full Elasticsearch.yml file

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.0.2.15
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.type: single-node

#
# discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
# cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 07-03-2022 00:10:42
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ubuntu"]

# Allow HTTP API connections from localhost and local networks
# Connections are encrypted and require user authentication
http.host: [_local_, _site_]

# Allow other nodes to join the cluster from localhost and local networks
# Connections are encrypted and mutually authenticated
#transport.host: [_local_, _site_]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

1 Like

java.lang.IllegalArgumentException: setting [cluster.initial_master_nodes] is not allowed when [discovery.type] is set to [single-node]

I don't get how this is coming up since I added discovery.type: single-node

This is coming up because you added discovery.type: single-node. Elasticsearch adds cluster.initial_master_nodes on first run as part of the security auto-configuration process. The message is simply telling you that you cannot set both settings.

1 Like

I get that but looking at the yml file I included, isn't cluster.initial_master_nodes commented out?

No, it’s not commented out

welp, got it working finally. My dumbass changed the password to elastic using passwd instead of

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
1 Like