Persistent keystore errors - guidance for new install please

I am trying to install elasticsearch both to learn how to use it - and as I need to try to make a working system too.

I first tried installing with notes from here over a year ago, but right from the start I had issues with sha security. I managed to find ways around it on youtube videos, but still struggled with conflicting information.

Initially, can I just ask a few questions please:- Each node will be on an HPE DL360 G9 and using RHEL Version 9 (unless told to install 8 instead). In either case, it will be updated.

  1. Is it best to create a normal user login called elasticsearch to install / configure elasticsearch. Or should a different account name / permission level be used?

  2. My main issues on my last try were persistent keystore errors. I tried deleting and regenerating, switched from P12 to PEM, reset permissions, reset ownership and so on. I even tried using AI to help sort it - and it too, was keen to have me delete / rebuild certificates. The passswords were NOT corrupted as I could extract keys etc when I had entered the required password / PEM authentication.

Examining the log would invariably complain about keystore password issues - even when briefly changing everything to an "idiot proof" one (or so I thought) of just DFGHJKL.

Is there a "common fault", that noobs like myself tend to make here?

As I tried so many different things, I am now going to do a clean install of RHEL and want to try again.

I would value any thoughts and ideas on "good combinations" - eg: RHEL8 + ES8.4 or RHEL9+ES9.?

Also, any comments on whether to do this logged into RHEL as "elasticsearch", root (doubt it) or "other" please.

I have asked in the past if there was a simple FAQ for installations - or even a kind of "template" where you define what nodes you want and it makes the hosts, yml and instances etc for you. But I appreciate that might be too easy!

Any help welcome please - going around in circles at the moment.

My $0.02

As elasticsearch is all run within a JVM, and a bundled one is provided, the specific RHEL (or Ubuntu or ..) version makes not much difference in my view. Well, more important would be any other local factors/policies within your organization. There's certainly no real "correlation" between RHEl 8 / ES 8.x and RHEL 9 / ES 9.x, see

for the specific mappings.

If you are starting from zero (again) I'd suggest use latest versions of both, RHEL 9.x +ES 9.x.

If you use the yum repository, it creates the required (local) user on the RHEL system for you, and again I'd see no reason to not let it do so.

Don't get confused between the linux user (with a uid, gid, etc, as defined in /etc/passwd) and the "users" within the Elasticsearch cluster itself, e.g. the "elastic" cluster user that is created during initial cluster setup.

As to the keystore issues, well I dont understand your problem description well enough to answer.

For installation, I've followed the ES docs before and personally I've found them pretty clear. Sadly a recent documentation update has been a step in the wrong direction, but you can start from here:

You might want to spin up a virtual machine or three to "practice".

PS: From your profile, "A rusty programmer ...", to the younger generation, might suggest you do your thing with Rust. And Rust and 6502 assembler (yes, I'm of the same vintage) are about as far apart as things can get.

1 Like

Thank you for that - which is most helpful - and I will read the suggested guide shortly.

There are no organizational policies in place here until elasticsearch is set up - the policy will then basically be along the lines of "keep the cluster happy, anything else needs to fit around that". The advantage of being on totally clean boxes I thought was "elastic can call the shots, everything else will need to fit around that".

I do take on board the difference between elastic cluster and the /etc/passwd users. I just wasn't sure (and did not see it in any documentation that I have look at), as to what I should be logged in with to do the install.

I have to be logged in as something to install elasticsearch - and just wondered if this should be root, <any user, normal privs), or a specific name. I figured default permissions might be influenced by what I was logged in with - and wondered if there was an "obvious choice" as to which linux user should do the install.

Useful comment on the JVM too, thank you - as my guide had specifically suggesting I should run update on RHEL and then install JAVA-17...

dnf update -y
dnf install -y java-17-openjdk wget unzip vim

...while others have suggested that I should NOT install java as it may wish a different version.

I just wish there was a simple "Installation FAQ for Noobs" - though hopefully the guide you linked to, will help me on my way.

Good catch on the "rusty programmer", I will change that. I simply meant as you surmised I think, that I was a happy chap with 6502 and the delights of Basic (and a macro based version of it in C++), but I have no knowledge of Rust.

Thank you for your time, it is appreciated!

Modern Linux distros discourage direct use of the root user, and usually at OS installation time create a user with admin (via sudo) privileges. Old school like me might not be a massive fan of sudo, but I've given up and just accept thats the way it is now.

When you come to install Elasticsearch from a yum repo, you will see output like:

Running transaction
  Preparing        :                                                                                                1/1
  Running scriptlet: elasticsearch-9.0.1-1.x86_64                                                                   1/1
Creating elasticsearch group... OK
Creating elasticsearch user... OK

where it creates the relevant users who will own the elasticsearch processes.

[kevin@rhel9x ~]$ id elasticsearch
uid=979(elasticsearch) gid=978(elasticsearch) groups=978(elasticsearch)

In RHEL case you will also see various output like:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : whateveritischosenrandomly
...

As it happens my RHEL 9 has this version of openjdk installed already, probably by some by some dependency.

[kevin@rhel9x ~]$ java -version
openjdk version "1.8.0_442"
OpenJDK Runtime Environment (build 1.8.0_442-b06)
OpenJDK 64-Bit Server VM (build 25.442-b06, mixed mode)

But elasticsearch ships with and uses it's own by default, and in your shoes I would not change this.

[kevin@rhel9x ~]$ /usr/share/elasticsearch/jdk/bin/java -version
openjdk version "24" 2025-03-18
OpenJDK Runtime Environment (build 24+36-3646)
OpenJDK 64-Bit Server VM (build 24+36-3646, mixed mode, sharing)

I like using VMs to learn. Do a base OS install, document what you did, make a snapshot. Then install say elasticsearch and associated needed tools, document your setup process, making notes, until you are happy. Then revert everything to your base snapshot and see if your notes are good enough to go from zero to hero. if so, you are done. If not, improve your notes. Iterate til done. Use that process to instal for real on the real hardware.

1 Like

Fantastic feedback, thank you.

I have installed VMs on a little linux laptop I have, but had not installed it on a project servers.

I think that is my next little task!

Makes excellent sense, and I will start to work with that today. The snapshot idea will make "starting over" a "while I pop the kettle on" job instead of a 1 hour job.

Originally I just thought "utterly clean install, nothing else loaded - should install great!".... (I can picture you laughing :slight_smile: )

Again, many thanks for offering simple, practical guidance, it really helps!

Update:

This is crazy but I am struggling now to even install VMWARE. All of the guides I can find no longer work since Broadcom took over BUT I have managed to download it.

My issue is that it wants to compile various things before it can run - and although I have done the updates and updated the headers, it now complains because although I have the LATEST headers, it is seeking OLDER ones - and I am not sure how to either download old headers for it to play with OR make it use new headers.

I would understand lots of teething issues with botched installs on heavily modified and tweeked systems - but this much drama to install either Elasticsearch or VMWARE on utterly clean RHEL installations does seem daft.

I don't seem to recall this much "fun" 40 years ago with Unix and Xenix. You extracted, ran, it installed and everything just seemed to work.

Anyway, is there a simple way to either tell VMWARE to use the LATEST headers OR for me to be able to download the OLD ones for it please?

I mean I don't need VMWARE other than to try to find different ways to make a completely clean installation of Elastic worth on a completely clean RHEL server - but it did seem as though this might allow me to easily start over each time ES failed again.

Is there an issue with RHEL where if I simply switch to something like Rocky Linux (or any other version), a clean install of ES on a clean install of linux is not likely to end up with security issue about corrupted keystores - or complain about security being set when I have specifically switched off ALL security in the yml.

I realise it must be me doing something wrong, but using Google just finds a lot of others with remarkably similar issues - but with no posted resolutions!

Bring back Dos 5 (or 6.22) :slight_smile:

I am a little lost. I think you are over-complicating things maybe based on reading too much.

To install single-node elasticsearch instance onto a fresh RHEL install should be 1) import a key, 2) add a yum repo, 3) install the package - yum/dnf does all the work. To make a cluster would require a little more work, but not much.

The subject mentions "persistent keystore errors", and you mentioned again "corrupted keystones", but if I've understood correctly these were problems last time, not currently? If thats not the case, please share/show more information incl which errors you are getting now?

As for installing VMware, well it's a while since I used it but I guess it needs build kernel modules. I use VirtualBox, but there's not much difference really, could use virt-install/virt-manager too. There's really too many options for virtualisation nowadays :wink:

I just so happened to install a RHEL 10 (test) system yesterday, let give it a go, not officially supported yet but ... will be much the same as on RHEL 8/9. I will follow:

Note it seemed to work, with only a minor whine around var/run which is now a link to /run

/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.

which is not important for here.

Here's the full process, my user (kevin) has sudo rights.

[kevin@rhel10x1 ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[kevin@rhel10x1 ~]$ sudo vi /etc/yum.repos.d/elastic.repo
< pasted the contents shown below >
[kevin@rhel10x1 ~]$ sudo cat /etc/yum.repos.d/elastic.repo
[elasticsearch]
name=Elasticsearch repository for 9.x packages
baseurl=https://artifacts.elastic.co/packages/9.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

[kevin@rhel10x1 ~]$ id elasticsearch
id: ‘elasticsearch’: no such user

[kevin@rhel10x1 ~]$ sudo dnf install --enablerepo=elasticsearch elasticsearch -y
Updating Subscription Management repositories.
Last metadata expiration check: 0:11:58 ago on Mon Jun 16 21:06:05 2025.
Dependencies resolved.
========================================================================================================================
 Package                        Architecture            Version                    Repository                      Size
========================================================================================================================
Installing:
 elasticsearch                  x86_64                  9.0.2-1                    elasticsearch                  620 M

Transaction Summary
========================================================================================================================
Install  1 Package

Total download size: 620 M
Installed size: 1.2 G
Downloading Packages:
elasticsearch-9.0.2-x86_64.rpm                                                          5.5 MB/s | 620 MB     01:52
------------------------------------------------------------------------------------------------------------------------
Total                                                                                   5.5 MB/s | 620 MB     01:52
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                1/1
  Running scriptlet: elasticsearch-9.0.2-1.x86_64                                                                   1/1
Creating elasticsearch group... OK
Creating elasticsearch user... OK

  Installing       : elasticsearch-9.0.2-1.x86_64                                                                   1/1
  Running scriptlet: elasticsearch-9.0.2-1.x86_64                                                                   1/1
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : UNMkFuZcZNH=QRg9q22F

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.

Installed products updated.

Installed:
  elasticsearch-9.0.2-1.x86_64

Complete!

[kevin@rhel10x1 ~]$ sudo systemctl start elasticsearch.service
[kevin@rhel10x1 ~]$ sudo systemctl status elasticsearch.service | cat
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; preset: disabled)
     Active: active (running) since Mon 2025-06-16 21:22:43 UTC; 24s ago
 Invocation: ab33179b9abc44ecafcb9f0a019d8d62
       Docs: https://www.elastic.co
   Main PID: 5205 (java)
      Tasks: 111 (limit: 10395)
     Memory: 1.3G (peak: 1.3G)
        CPU: 40.407s
     CGroup: /system.slice/elasticsearch.service
             ├─5205 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.type=rpm "-Des.java.type=bundled JDK" -cp "/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/cli-launcher/*" org.elasticsearch.launcher.CliToolLauncher -p /var/run/elasticsearch/elasticsearch.pid --quiet
             ├─5267 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=CLDR -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24 -Des.distribution.type=rpm "-Des.java.type=bundled JDK" --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core --enable-native-access=ALL-UNNAMED --illegal-native-access=deny -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log -Des.entitlements.enabled=true -XX:+EnableDynamicAgentLoading -Djdk.attach.allowAttachSelf=true --patch-module=java.base=lib/entitlement-bridge/elasticsearch-entitlement-bridge-9.0.2.jar --add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-7540940937904985010 --add-modules=jdk.incubator.vector -Dorg.apache.lucene.store.defaultReadAdvice=normal -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m" -Xms852m -Xmx852m -XX:MaxDirectMemorySize=446693376 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path /usr/share/elasticsearch/lib --add-modules=jdk.net --add-modules=jdk.management.agent --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch
             └─5287 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jun 16 21:21:56 rhel10x1 systemd[1]: Starting elasticsearch.service - Elasticsearch...
Jun 16 21:22:43 rhel10x1 systemd[1]: Started elasticsearch.service - Elasticsearch.

[kevin@rhel10x1 ~]$ EUSER=elastic EPASS="UNMkFuZcZNH=QRg9q22F"
[kevin@rhel10x1 ~]$ curl -sku "${EUSER}:${EPASS}" https://localhost:9200 | jq -r .
{
  "name": "rhel10x1",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "_wrTbkCRQauLMU8ZfYXtZA",
  "version": {
    "number": "9.0.2",
    "build_flavor": "default",
    "build_type": "rpm",
    "build_hash": "0a58bc1dc7a4ae5412db66624aab968370bd44ce",
    "build_date": "2025-05-28T10:06:37.834829258Z",
    "build_snapshot": false,
    "lucene_version": "10.1.0",
    "minimum_wire_compatibility_version": "8.18.0",
    "minimum_index_compatibility_version": "8.0.0"
  },
  "tagline": "You Know, for Search"
}

[kevin@rhel10x1 ~]$ curl -sku "${EUSER}:${EPASS}" https://localhost:9200/_cat/indices
[kevin@rhel10x1 ~]$ curl -sku "${EUSER}:${EPASS}" "https://localhost:9200/_cat/indices?index=.*"
green open .security-7                                                   h3oQBa82SWqOwGEayi3T_g 1 0 30 0 40.9kb 40.9kb 40.9kb
green open .ds-ilm-history-7-2025.06.16-000001                           OcN-xcEiRniz30GpGO5Ouw 1 0  3 0  9.7kb  9.7kb  9.7kb
green open .ds-.logs-elasticsearch.deprecation-default-2025.06.16-000001 497jow5-S_qExb7LJzjNxA 1 0  1 0 10.2kb 10.2kb 10.2kb

[kevin@rhel10x1 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="10.0 (Coughlan)"
ID="rhel"
ID_LIKE="centos fedora"
VERSION_ID="10.0"
PLATFORM_ID="platform:el10"
PRETTY_NAME="Red Hat Enterprise Linux 10.0 (Coughlan)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:10::baseos"
HOME_URL="https://www.redhat.com/"
VENDOR_NAME="Red Hat"
VENDOR_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 10"
REDHAT_BUGZILLA_PRODUCT_VERSION=10.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="10.0"

[kevin@rhel10x1 ~]$ head -1 /proc/meminfo
MemTotal:        1746136 kB

[kevin@rhel10x1 ~]$ egrep ^proc /proc/cpuinfo
processor	: 0
processor	: 1
[kevin@rhel10x1 ~]$

[kevin@rhel10x1 ~]$ sudo ls -ld /var/run /run
drwxr-xr-x. 38 root root 1140 Jun 16 21:21 /run
lrwxrwxrwx.  1 root root    6 Apr 23 14:35 /var/run -> ../run

[kevin@rhel10x1 ~]$ id elasticsearch
uid=990(elasticsearch) gid=990(elasticsearch) groups=990(elasticsearch)

[kevin@rhel10x1 ~]$ ps -uelasticsearch -opid,command | cat
    PID COMMAND
   5205 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.type=rpm -Des.java.type=bundled JDK -cp /usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher -p /var/run/elasticsearch/elasticsearch.pid --quiet
   5267 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=CLDR -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24 -Des.distribution.type=rpm -Des.java.type=bundled JDK --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core --enable-native-access=ALL-UNNAMED --illegal-native-access=deny -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log -Des.entitlements.enabled=true -XX:+EnableDynamicAgentLoading -Djdk.attach.allowAttachSelf=true --patch-module=java.base=lib/entitlement-bridge/elasticsearch-entitlement-bridge-9.0.2.jar --add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-7540940937904985010 --add-modules=jdk.incubator.vector -Dorg.apache.lucene.store.defaultReadAdvice=normal -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Xms852m -Xmx852m -XX:MaxDirectMemorySize=446693376 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path /usr/share/elasticsearch/lib --add-modules=jdk.net --add-modules=jdk.management.agent --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch
   5287 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
2 Likes

Thank you. I will try this.

Possibly the mistake I have made is that when I explained to folks I wanted to setup a cluster, it was suggested to edit the yml files giving details of master intitial nodes etc - and also the instances file so each node knows the IP addresses of the others.

As such I created custom yml files (example of one below) so I could generate the security certs in one go - and then copy them to the other nodes.

I am very happy to just create a single node - which I hope to be able to get working tonight - but then, I am just concerned that making it "make friends with other nodes" will then be more difficult.

Anyway, I will have a go installing EXACTLY as you suggest - which I suspect will work, (and Thank you either way for your time and kindness).

My concern will then be how to make them "find each other", but I guess first step first :slight_smile:

Again, thank you for your time, I shall do a completely clean install now with RHEL 9 and try with your guide.

I guess you forgot to include the yml file.

Good luck with the RHEL/elasticsearch installation.

[ Just FYI, I can tell you for 90%+ of the queries / questions I try to answer on here, I use a single elasticsearch and single kibana instance (both running on my M1 Mac Mini) to check stuff. It also houses some actual data and an irregularly-taken (do as I say, not as I do!) snapshot repo, the repo backed up via Time Machine (I really should use another USB drive for a more resilient home for my repo). I have a SFF miniPC (cost me 250 euro, I'm Scottish!) system hanging off the back of my monitor (literally) on which I have Ubuntu, and run VirtualBox, docker, podman, minikube, ... eg the RHEL10 was a quick install using virt-install. Moral of all of this is nowadays you can do a lot, and learn a lot, with relatively little. ]

Apologies, you are quite right, I ended up copying the yml into my notes instead of on here.

This is what I was using - though there were other iterations tried.:-

cluster.name: testcluster
node.name: es1
network.host: 172.22.22.131
discovery.seed_hosts: ["172.22.22.131", "172.22.22.132", "172.22.22.133"]
cluster.initial_master_nodes: ["es1", "es2", "es3"]
node.roles: [ master, data, remote_cluster_client ]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/es1.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/es1.p12
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

For now though, I have not touched the elasticsearch.yml or instances.yml

Your comments on using VM for doing various test is excellent - and I have managed to install it on a Mint Cinnamon laptop - but struggling to install vmware on an HP DL360 G9.

I will respond when I have ES installed and stable!

A short update before bed.

Thank you so much for your patient assistance, I now have a working single node, I will make another on Friday and see if I can make them "see" each other.

Thanks for update. I suggest also to install lsof on all 3 nodes:

sudo dnf install lsof -y

and then

sudo lost -i :9200

and

ps -uelasticsearch -opid= | xargs -n1 sudo lsof -nPp | fgrep TCP

on the single node that is installed/working.

And you should have close to the "vendor default" elasticsearch,yml on that node too

sudo cat /etc/elasticsearch/elasticsearch.yml | grep -Ev \^# | egrep . | cat

The gotchas you might need to avoid are elasticsearch only listening on loopback interfaces by default, the lsof commands can show you this, and messing about with discovery and initial-nodes settings that some guides suggest.

Assuming you have your enrollment token from the output when node1 was setup, and you can always regenerate one anyways, on other nodes it should be as simple as

sudo /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token aVeryLongToken==

You can use same token for all other nodes.

IMHO once you have all the nodes in the cluster, you should tidy up/align the elasticsearch,yml files, my own 3-node cluster has ended up with following settings:

[kevin@rhel10x3 ~]$ sudo cat /etc/elasticsearch/elasticsearch.yml | grep -Ev \^# | egrep . | cat
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
discovery.seed_hosts: ["rhel10x1:9300", "rhel10x2:9300","rhel10x3:9300"]
http.host: 0.0.0.0
transport.host: _site_

(I had already added the hostname entries in /etc/hosts on all nodes to avoid remembering the local IP addresses)

How did you get on with the next steps?

Hello.

I now have three isolated stand-alone nodes which all load up independently - which I would like to work together as the initial masters and the "always on set". These boxes will not run anything other than Elasticsearch (other than updates / etc)

I also have one further node which is on a shared box. It will often be switched off, but when on, it may need to be able to join the cluster, as its job is to prepare data and then ingest it.

I am trying to do as much as I can without pestering folks like yourself - so working out at the moment things like "How to tell elasticsearch that it needs to use network port #2 to reach the rest of the cluster" and so on. Once I have done that though, I think I am going to need a little guidance as I suspect once I start editing the yml files, I may end up with some more "fun".

So far so good though - as I have four boxes which all seem to be happy to run Elasticsearch - even if, at the moment, they do not know the other nodes exist.

Thanks for update. You seem on right trajectory.

I simply would not include/add this to the cluster, based on what you wrote there. Preparing and ingesting data can be done from outside the cluster footprint. Others might disagree on this.

If your eventual cluster yml file has more than a dozen settings/lines, it’s likely bloated.

1 Like

@GuyMark did you reach Nirvana ?

Not yet!

But I have managed to get pretty much all of Friday set aside to have another go at making them talk to each other.

I noted what you said about not using ingestion nodes to ingest data and I have not managed to find much info on that - but as I have not yet managed to get the initial three nodes happily talking, I have not spent too much time on that yet.

If I cannot sort on Friday, I will give specific issues that I am struggling with on the basis that it is likely to be something daft.

Some progress :slight_smile: I was unduly concerned about some error messages which are apparently ignorable when running:-

$ ps -uelasticsearch -opid= | xargs -n1 sudo lsof -nPp | fgrep TCP

I am now using RHEL 10 and that seems to have lsof installed by default.

However when I try the command:-

$ sudo /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token mytoken

I keep getting the following.

ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only., with exit code 80

Other than making sure it installed, nodes 2 and 3 have never been started before, nor do they have any data on them.

I have re-installed nodes 2 and 3 a few times but cannot find a way to override this error.

I have tried Google - but following that was what made things "more fun" in the first place.

I have not yet edited the yml file, as you suggested to do that NEXT. So I am figuring the defaults will help it find each other (all servers have fixed IP and are connected to a switch that has nothing else connected other than PORT 4 of each box).

If you think I should edit the yml file FIRST before trying again, will happily do so, but thought I would query first.

Thank you for your support so far, it has been extremely helpful.

Yeah, it's not clear to me where you are with nodes 2/3. You might think elasticsearch was never started on the nodes, but ... might not be true.

What do you get for this command on all nodes?

sudo ls /var/lib/elasticsearch/

I get

$ sudo ls /var/lib/elasticsearch/
_state	indices  node.lock  nodes  snapshot_cache

because my (3-node) cluster has been running for a while.

The ps command is just too heck the port is open (on 1st node), this variant is easier and should return something similar on your system

[kevin@rhel10x1 ~]$ sudo lsof -Pni :9200
COMMAND  PID          USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    1658 elasticsearch  572u  IPv6  12423      0t0  TCP *:9200 (LISTEN)

Important is that its listening on a non-loopback interface.

well, on nodes 2/3 thats fine. But whats on node1?

Mine (now that I have 3-node cluster) is:

[kevin@rhel10x1 ~]$ sudo grep -E '^[^#].*' /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
discovery.seed_hosts: ["rhel10x1:9300", "rhel10x2:9300","rhel10x3:9300"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
http.host: 0.0.0.0
transport.host: _site_
xpack.monitoring.elasticsearch.collection.enabled: true
xpack.monitoring.collection.enabled: true

yours would not have the same settings for discovery.seed_hosts, but otherwise should be pretty similar.