Failed to start Elasticsearch

Elastic Stack Elasticsearch
1

Hi everyone, I'm new here ! :ok_woman:
I just finished set up basic security on my server (1VM with : Elasticsearch, 1 node, Kibana).

I ran those commands :

./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

And now Elasticsearch.service won't start.

Here is my elasticsearch.yml :

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: mycluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.xxxxxxx
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.type: single-node
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 17-10-2023 09:13:53
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/ims-elastic-certificates.p12
  truststore.path: certs/ims-elastic-certificates.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["node1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Here you have status elasticsearch.service :

× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-10-17 15:35:48 UTC; 6s ago
       Docs: https://www.elastic.co
    Process: 103854 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, st>
   Main PID: 103854 (code=exited, status=1/FAILURE)
        CPU: 20.939s

Oct 17 15:35:36 srv-elk-ims-2 systemd[1]: Starting Elasticsearch...
Oct 17 15:35:38 srv-elk-ims-2 systemd-entrypoint[103854]: Oct 17, 2023 3:35:38 PM sun.util.locale.provider.LocaleProviderAdapter <clin>
Oct 17 15:35:38 srv-elk-ims-2 systemd-entrypoint[103854]: WARNING: COMPAT locale provider will be removed in a future release
Oct 17 15:35:48 srv-elk-ims-2 systemd-entrypoint[103854]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elas>
Oct 17 15:35:48 srv-elk-ims-2 systemd-entrypoint[103854]: ERROR: Elasticsearch exited unexpectedly, with exit code 1
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: Failed to start Elasticsearch.
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: elasticsearch.service: Consumed 20.939s CPU time.

And here the journalctl -xeu elasticsearch.service :

 A start job for unit elasticsearch.service has begun execution.
░░
░░ The job identifier is 10795.
Oct 17 15:35:38 srv-elk-ims-2 systemd-entrypoint[103854]: Oct 17, 2023 3:35:38 PM sun.util.locale.provider.LocaleProviderAdapter <clin>
Oct 17 15:35:38 srv-elk-ims-2 systemd-entrypoint[103854]: WARNING: COMPAT locale provider will be removed in a future release
Oct 17 15:35:48 srv-elk-ims-2 systemd-entrypoint[103854]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elas>
Oct 17 15:35:48 srv-elk-ims-2 systemd-entrypoint[103854]: ERROR: Elasticsearch exited unexpectedly, with exit code 1
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit elasticsearch.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: Failed to start Elasticsearch.
░░ Subject: A start job for unit elasticsearch.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit elasticsearch.service has finished with a failure.
░░
░░ The job identifier is 10795 and the job result is failed.
Oct 17 15:35:48 srv-elk-ims-2 systemd[1]: elasticsearch.service: Consumed 20.939s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit elasticsearch.service completed and consumed the indicated resources.

Thank you in advance for your help ... :pray:

Betty

2

have you had a look at /var/log/elasticsearch/<latest_log_file>.
This is also suggested in your outputs :wink:

1 Like
3

Hi sholzhauer, thanks for your reply :slightly_smiling_face:

I looked in the /var/log/elasticsearch/mycluster.log and then deleted the discovery.type=single-node in the yml

Elasticsearch.service has restarted but now I'm facing the same problem (elasticsearch.service won't start) after that :

./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

Here is what the log is saying :

[2023-10-17T18:10:10,736][ERROR][o.e.b.Elasticsearch      ] [node1] fatal exception while booting Elasticsearch
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read config>
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:618) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1429) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1707) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:614) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:159) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:502) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:342) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$16(Node.java:738) ~[elasticsearch-8.10.4.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:261) ~[elasticsearch-8.10.4.jar:?]
at org.elasticsearch.node.Node.lambda$new$16(Node.java:738) ~[elasticsearch-8.10.4.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:261) ~[elasticsearch-8.10.4.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
        at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:754) ~[elasticsearch-8.10.4.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:338) ~[elasticsearch-8.10.4.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234) ~[elasticsearch-8.10.4.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234) ~[elasticsearch-8.10.4.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72) ~[elasticsearch-8.10.4.jar:?]
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticse>
        at org.elasticsearch.common.ssl.SslFileUtil.ioException(SslFileUtil.java:56) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:98) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:477) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1228) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:616) ~[?:?]
        ... 23 more
Caused by: java.io.IOException: keystore password was incorrect
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2097) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1500) ~[?:?]
at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
        at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]

I shown and reseted the password of xpack.security.http.ssl.keystore.secure_password but it won't start again.

I guess I'm missing something...

4

The password you checked/fixed is not the one that is reported in the error message.
You changed: xpack.security.http.ssl.keystore.secure_password
But the error seems to be referring to: xpack.security.transport.ssl.truststore.secure_password

(It's hard to tell because you've cut of the end of your log messages)

1 Like
5

Hi TimV thanks for your answer !

I don't understand the error cause I don't change the password after putting it.

I recently tried to use the same password for :

xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
xpack.security.http.ssl.keystore.secure_password

But didn't work either.

Maybe is there another password linked to one of those ?

Here you have my logs (sorry idk how to show the end of log message) :

[2023-10-18T06:29:36,508][ERROR][o.e.b.Elasticsearch      ] [node1] fatal exception while booting Elasticsearch
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/ims-elastic-certificates.p12] - this is usually caused by an incorrect password; (a keystore password was provided)

Does the chmod as to do something with that ? Here is my configuration :

/etc/elasticsearch/certs# ls -l /etc/elasticsearch/

drwxr-x--- 2 root elasticsearch  4096 Oct 17 17:42 certs
-rw-rw---- 1 root elasticsearch  1042 Oct  5 10:23 elasticsearch-plugins.example.yml
-rw-rw---- 1 root elasticsearch   517 Oct 18 07:11 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch  4090 Oct 17 17:16 elasticsearch.yml
-rw-r----- 1 root elasticsearch  4058 Oct 17 09:22 elasticsearch.yml.backup
-rw-r----- 1 root elasticsearch  4047 Oct 17 12:46 elasticsearch.yml.backup1
-rw-rw---- 1 root elasticsearch  2799 Oct 17 14:13 jvm.options
-rw-r----- 1 root elasticsearch  2781 Oct 17 14:11 jvm.options.backup
drwxr-s--- 2 root elasticsearch  4096 Oct  5 10:23 jvm.options.d
-rw-rw---- 1 root elasticsearch 17969 Oct  5 10:23 log4j2.properties
-rw-rw---- 1 root elasticsearch   473 Oct  5 10:23 role_mapping.yml
-rw-rw---- 1 root elasticsearch   197 Oct  5 10:23 roles.yml
-rw-rw---- 1 root elasticsearch     0 Oct 17 10:10 users
-rw-rw---- 1 root elasticsearch     0 Oct 17 10:10 users_roles

Thanks in advance for your reply...

6

Doest the password for xpack.security.transport.ssl.keystore.secure_password and xpack.security.http.ssl.keystore.secure_password has to be the same ?

I still get those errors :

elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/ims-elastic-certificates.p12]


Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/ims-elastic-certificates.p12] - this is usually caused by an incorrect password; (a keystore password was provided)

Please send help, this is so confusying haha

7

Hi @Tybe_sacha

Exactly what version are you installing?

And exactly what was the initial command you installed with?

All of this should have been taken care of automatically if you're using 8.X..

I'm confused with you're actually trying to accomplish. Why are you trying to manually set these passwords?

I would start over and let the automatic configuration run and then look how it set up works afterwards. That would be my suggestion if you're just getting started

8

Hi Stephen,

Here are my installation commands in the order (I tried with a folder first to install elastic cause my server will have to be install without internet) :

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
/source/elasticsearch_8.10.3_amd64.deb
sudo dpkg -i elasticsearch_8.10.3_amd64.deb

sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install kibana
sudo apt update && sudo apt install elasticsearch

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install kibana
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service

Elasticsearch and Kibana are both now on 8.10.4

I succeeded to configure basic setup from the official site and then was trying to set up basic security plus HTTPS.
After doing those commands, Elasticsearch won't start :

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

unzip /usr/share/elasticsearch/elasticsearch-ssl-http.zip

sudo mv /etc/elasticsearch/certs/http.p12 /etc/elasticsearch/certs/http.p12.old
sudo cp elasticsearch/http.p12 /etc/elasticsearch/certs/
chown root:elasticsearch http.p12
chmod 660 http.p12

./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

So you'd suggest to re-install Elasticsearch ? On my actual server ? Or just on another clean server ?

Betty

9

Hi,

You could try to verify if your keystore setup is actually correct, maybe rerun the keystore creation with simpler passwords to test the setup, remember to take notes and secure everything afterwards.

What i suggest to you is to check your existing keystores and try to test with java keytools if you can list and dump keys using your password.

This might just be typos and miss understand :wink:

10

Hi grumo35,

Thank for your answer :slight_smile:

Here is what I've done for the last hours :

My logs were saying :

Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/ims-elastic-certificates.p12] - this is usually caused by an incorrect password; (a keystore password was provided)

So I ran :

./bin/elasticsearch-keystore has-passwd
keystore is not password-protected

./bin/elasticsearch-keystore list
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

Maybe I wasn't refering to the good keystore but then I add a password and could list the files :

./bin/elasticsearch-keystore list
Enter password for the elasticsearch keystore :
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

So even before add the password I could list those files...
I'm afraid I applied to many changes or just be so lost now :rofl:

11

Elasticsearch HTTPS was already setup (it is setup as default)... why were you re-doing? ...Its ok I am just trying to understand...

Were you upgrading or fresh install?

12

I see you used the elastic keytool but what about the file itself ?

If you setup a password this means the file is encrypted and you cannot read keys or certificates inside.

Please try the following or try to find a way to "open such files" and see if the provided password works

openssl pkcs12 -nokeys -info \
    -in </path/to/file.p12> \
    -passin pass:<p12's password>
13

Haha no problem, I enjoy learning and actually this is the first I have to ask for help in a forum so I'm glad to find you :smile:

So trying to make it clear I follow this video first to : https://www.youtube.com/watch?v=kkrLanotz1I&t=229s

And then this one : https://www.youtube.com/watch?v=w3taEk9tAQ4

This guy is referring to the official website so I guess it was ok for me...

Elastic was 8.10.3 fresh install
Kibana was 8.10.4 fresh install
Then I upgrade Elastic to 8.10.4
14

I setup a password for each step of configuration and kept it in my Keepass :pensive:...

openssl pkcs12 -nokeys -info -in /etc/elasticsearch/certs/ims-elastic-certificates.p12

Enter Import Password:
MAC: sha256, Iteration 10000
MAC length: 32, salt length: 20
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 10000, PRF hmacWithSHA256
Certificate bag
15

Me... I would recommend our official docs as the best starting point.

I think (as @grumo35 suggests) that perhaps you have more than one Elastic keystore or that the passwords you entered in the certs do not match what you put in the Elastic keystore

I think you have 2 keystores 1 in the right place ... 1 somewhere else..

The right place should be /etc/elasticsearch

Try the full path ... the docs show the relative commands for the tar.gz install for the package install

/etc/elasticsearch# ls -lrt
total 84
-rw-rw---- 1 root elasticsearch   197 Jun  8  2022 roles.yml
-rw-rw---- 1 root elasticsearch   473 Jun  8  2022 role_mapping.yml
drwxr-s--- 2 root elasticsearch  4096 Jun  8  2022 jvm.options.d
-rw-rw---- 1 root elasticsearch  1042 Jun  8  2022 elasticsearch-plugins.example.yml
-rw-rw---- 1 root elasticsearch 17417 Sep 14  2022 log4j2.properties
-rw-rw---- 1 root elasticsearch  2617 Feb 13  2023 jvm.options
-rw-r----- 1 root elasticsearch  4294 Mar 17  2023 elasticsearch.yml.work
-rw-r--r-- 1 root elasticsearch   141 Mar 17  2023 trust.yml
drwxr-sr-x 5 root elasticsearch  4096 Mar 17  2023 newcerts
-rw-r--r-- 1 root elasticsearch  1657 May 10 03:27 setup.txt
-rw-rw---- 1 root elasticsearch  4808 Jun 15 20:02 elasticsearch.yml
drwxr-xr-x 2 root elasticsearch  4096 Jun 15 20:20 certs
-rw-rw---- 1 root elasticsearch   252 Jun 15 20:59 users
-rw-rw---- 1 root elasticsearch    79 Jun 15 20:59 users_roles
-rw-rw---- 1 root elasticsearch   327 Oct 18 14:45 elasticsearch.keystore <!---- HERE :)

So run the commands

/etc/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-keystore list
autoconfiguration.password_hash
keystore.seed
my.setting
root@stephenb-es-8-test:/etc/elasticsearch#
16

Hello everyone,

I really appreciate your help, thank you !

I'm looking for the different keystore but not sure if I find it :

root@srv-elk-ims-2:/home/imsadmin# ls -l /usr/share/elasticsearch/bin
total 3208
-rwxr-xr-x 1 root root     101 Oct 11 22:10 elasticsearch
-rwxr-xr-x 1 root root     376 Oct 11 22:10 elasticsearch-certgen
-rwxr-xr-x 1 root root     376 Oct 11 22:10 elasticsearch-certutil
-rwxr-xr-x 1 root root     674 Oct 11 22:10 elasticsearch-cli
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-create-enrollment-token
-rwxr-xr-x 1 root root     352 Oct 11 22:10 elasticsearch-croneval
-rwxr-xr-x 1 root root    2338 Oct 11 22:10 elasticsearch-env
-rwxr-xr-x 1 root root    2595 Oct 11 22:10 elasticsearch-env-from-file
-rwxr-xr-x 1 root root      84 Oct 11 22:10 elasticsearch-geoip
*-rwxr-xr-x 1 root root      87 Oct 11 22:10 elasticsearch-keystore*
-rwxr-xr-x 1 root root      55 Oct 11 22:10 elasticsearch-node
-rwxr-xr-x 1 root root     172 Oct 11 22:10 elasticsearch-plugin
-rwxr-xr-x 1 root root     376 Oct 11 22:10 elasticsearch-reconfigure-node
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-reset-password
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-saml-metadata
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-service-tokens
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-setup-passwords
-rwxr-xr-x 1 root root      55 Oct 11 22:10 elasticsearch-shard
-rwxr-xr-x 1 root root     403 Oct 11 22:10 elasticsearch-sql-cli
-rwxr-xr-x 1 root root 3188571 Oct 11 22:10 elasticsearch-sql-cli-8.10.4.jar
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-syskeygen
-rwxr-xr-x 1 root root     353 Oct 11 22:10 elasticsearch-users
drwxr-xr-x 2 root root    4096 Oct 17 17:33 kibana
-rwxr-xr-x 1 root root     332 Oct 11 22:11 systemd-entrypoint

root@srv-elk-ims-2:/home/imsadmin# ls -l /etc/elasticsearch/
total 68
drwxr-x--- 2 root elasticsearch  4096 Oct 18 09:57 certs
-rw-rw---- 1 root elasticsearch  1042 Oct  5 10:23 elasticsearch-plugins.example.yml
*-rw-rw---- 1 root elasticsearch   518 Oct 18 12:57 elasticsearch.keystore*
-rw-rw---- 1 root elasticsearch  4124 Oct 18 13:50 elasticsearch.yml
-rw-r----- 1 root elasticsearch  4058 Oct 17 09:22 elasticsearch.yml.backup
-rw-r----- 1 root elasticsearch  4047 Oct 17 12:46 elasticsearch.yml.backup1
-rw-rw---- 1 root elasticsearch  2799 Oct 17 14:13 jvm.options
-rw-r----- 1 root elasticsearch  2781 Oct 17 14:11 jvm.options.backup
drwxr-s--- 2 root elasticsearch  4096 Oct  5 10:23 jvm.options.d
-rw-rw---- 1 root elasticsearch 17969 Oct  5 10:23 log4j2.properties
-rw-rw---- 1 root elasticsearch   473 Oct  5 10:23 role_mapping.yml
-rw-rw---- 1 root elasticsearch   197 Oct  5 10:23 roles.yml
-rw-rw---- 1 root elasticsearch     0 Oct 17 10:10 users
-rw-rw---- 1 root elasticsearch     0 Oct 17 10:10 users_roles
root@srv-elk-ims-2:/home/imsadmin#

The second one is the command to store the passwords in it right ?

Here are my 2 questions about the official documentation :

./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

I mean, how does it works to understand which password I need to store ? (CA, certificates, private key...)

Have a good day !

17

Additional question :

  • Why my keystore is elasticsearch.keystore and not "ims-elastic-certificates.p12" as it's mentionned ?
 keystore.path: certs/ims-elastic-certificates.p12
 truststore.path: certs/ims-elastic-certificates.p12

Or do I have to mentionned /etc/elasticsearch/elasticsearch.keystore ?

EDIT :
I've been working on finding both keystore :

root@srv-elk-ims-2:/etc/elasticsearch/certs# ls -l
total 36
-rw-rw---- 1 root elasticsearch  3620 Oct 18 09:43 http.p12
-rw-rw---- 1 root elasticsearch 10045 Oct 17 09:14 http.p12.old
-rw-rw---- 1 root elasticsearch  3620 Oct 17 17:42 http.p12.old1
-rw-rw---- 1 root elasticsearch  1915 Oct 17 09:14 http_ca.crt
-rw-rw---- 1 root elasticsearch  3596 Oct 17 13:20 ims-elastic-certificates.p12
-rw-rw---- 1 root elasticsearch  5822 Oct 17 09:14 transport.p12
root@srv-elk-ims-2:/etc/elasticsearch/certs# /usr/share/elasticsearch/bin/elasticsearch-keystore list
Enter password for the elasticsearch keystore :

ERROR: Provided keystore password was incorrect, with exit code 65

root@srv-elk-ims-2:/etc/elasticsearch/certs# ls -l /etc/elasticsearch/
total 68
drwxr-x--- 2 root elasticsearch  4096 Oct 18 09:57 certs
-rw-rw---- 1 root elasticsearch  1042 Oct  5 10:23 elasticsearch-plugins.example.yml
-rw-rw---- 1 root elasticsearch   518 Oct 18 12:57 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch  4124 Oct 18 13:50 elasticsearch.yml
-rw-r----- 1 root elasticsearch  4058 Oct 17 09:22 elasticsearch.yml.backup
-rw-r----- 1 root elasticsearch  4047 Oct 17 12:46 elasticsearch.yml.backup1
-rw-rw---- 1 root elasticsearch  2799 Oct 17 14:13 jvm.options
-rw-r----- 1 root elasticsearch  2781 Oct 17 14:11 jvm.options.backup
drwxr-s--- 2 root elasticsearch  4096 Oct  5 10:23 jvm.options.d
-rw-rw---- 1 root elasticsearch 17969 Oct  5 10:23 log4j2.properties
-rw-rw---- 1 root elasticsearch   473 Oct  5 10:23 role_mapping.yml
-rw-rw---- 1 root elasticsearch   197 Oct  5 10:23 roles.yml
-rw-rw---- 1 root elasticsearch     0 Oct 17 10:10 users
-rw-rw---- 1 root elasticsearch     0 Oct 17 10:10 users_roles

root@srv-elk-ims-2:/etc/elasticsearch/certs# cd /etc/elasticsearch/
root@srv-elk-ims-2:/etc/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-keystore list
Enter password for the elasticsearch keystore :
autoconfiguration.password_hash
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

So you were totally right !! Are my two keystores the followings ? :
/etc/elasticsearch/elasticsearch.keystore
and
/etc/elasticsearch/certs/ims-elastic-certificates.p12

So I'll have to follow "Update security certificate with a different CA" I guess : Update security certificates with a different CA | Elasticsearch Guide [8.10] | Elastic

18

Hi,

Everything is just ones and zeroes, you can check file with the command "file" or try to open them and see.

As i thought you need to have a better understanding of what your command is doing, and where the file are created / managed.

Just delete everything, start from fresh documentation !

It's probably just a typo anyway....

The link you provided is about trusting an internal CA, certificate chains can be quite complicated and simple if you take time to understand the small mechanism there is behind "Trust".

I think the proposed default setup is about self signed certificate. Not sure you should follow this.

19

Hi grumo35,

Yeah you right, I'm trying hard to understand all commands, and trusted certificated but I'm just gonna go from the beggining...

Can I juste delete those files ?
elasticsearch.keystore
ims-elastic-certificates.p12
ims-elastic-ca.p12

Or is there a way to delete and then start from zero ?

Thank you again for your kindly help :smile:

20

@Tybe_sacha

You are being confused by overloaded terms.

The first is a keystore that elastic uses to store secrets

The other is a certificate keystore

2 different things / with the same name ... unfortunately.