# Persistent keystore errors - guidance for new install please **URL:** \ **Category:** Elasticsearch\ **Created:** [May 27, 2025, 11:36pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620 "2025-05-27T23:36:57Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [May 27, 2025, 11:36pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/1 "2025-05-27T23:36:57Z")
I am trying to install elasticsearch both to learn how to use it - and as I need to try to make a working system too. I first tried installing with notes from here over a year ago, but right from the start I had issues with sha security. I managed to find ways around it on youtube videos, but still struggled with conflicting information. Initially, can I just ask a few questions please:- Each node will be on an HPE DL360 G9 and using RHEL Version 9 (unless told to install 8 instead). In either case, it will be updated. 1. Is it best to create a normal user login called elasticsearch to install / configure elasticsearch. Or should a different account name / permission level be used? 2. My main issues on my last try were persistent keystore errors. I tried deleting and regenerating, switched from P12 to PEM, reset permissions, reset ownership and so on. I even tried using AI to help sort it - and it too, was keen to have me delete / rebuild certificates. The passswords were NOT corrupted as I could extract keys etc when I had entered the required password / PEM authentication. Examining the log would invariably complain about keystore password issues - even when briefly changing everything to an "idiot proof" one (or so I thought) of just DFGHJKL. Is there a "common fault", that noobs like myself tend to make here? As I tried so many different things, I am now going to do a clean install of RHEL and want to try again. I would value any thoughts and ideas on "good combinations" - eg: RHEL8 + ES8.4 or RHEL9+ES9.? Also, any comments on whether to do this logged into RHEL as "elasticsearch", root (doubt it) or "other" please. I have asked in the past if there was a simple FAQ for installations - or even a kind of "template" where you define what nodes you want and it makes the hosts, yml and instances etc for you. But I appreciate that might be too easy! Any help welcome please - going around in circles at the moment. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [May 28, 2025, 3:17am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/2 "2025-05-28T03:17:04Z")
> [@GuyMark](#): > > I would value any thoughts and ideas on "good combinations" - eg: RHEL8 + ES8.4 or RHEL9+ES9.? My $0.02 As elasticsearch is all run within a JVM, and a bundled one is provided, the specific RHEL (or Ubuntu or ..) version makes not much difference in my view. Well, more important would be any other local factors/policies within your organization. There's certainly no real "correlation" between RHEl 8 / ES 8.x and RHEL 9 / ES 9.x, see > **[Support Matrix](https://www.elastic.co/support/matrix)** > > The tables below display platform and software configurations that are eligible for support under our > subscription offerings. Learn more about our > Support Policy and product End of Life poli... for the specific mappings. If you are starting from zero (again) I'd suggest use latest versions of both, RHEL 9.x +ES 9.x. If you use the yum repository, it creates the required (local) user on the RHEL system for you, and again I'd see no reason to not let it do so. Don't get confused between the linux user (with a uid, gid, etc, as defined in /etc/passwd) and the "users" within the Elasticsearch cluster itself, e.g. the "elastic" cluster user that is created during initial cluster setup. As to the keystore issues, well I dont understand your problem description well enough to answer. For installation, I've followed the ES docs before and personally I've found them pretty clear. Sadly a recent documentation update has been a step in the wrong direction, but you can start from here: > **[Install Elasticsearch with RPM | Elastic Docs](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm)** > > The RPM package for Elasticsearch can be downloaded from our website or from our RPM repository. It can be used to install Elasticsearch on any RPM-based... You might want to spin up a virtual machine or three to "practice". PS: From your profile, "A rusty programmer ...", to the younger generation, might suggest you do your thing with Rust. And Rust and 6502 assembler (yes, I'm of the same vintage) are about as far apart as things can get. ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [May 28, 2025, 10:47am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/3 "2025-05-28T10:47:54Z")
Thank you for that - which is most helpful - and I will read the suggested guide shortly. There are no organizational policies in place here until elasticsearch is set up - the policy will then basically be along the lines of "keep the cluster happy, anything else needs to fit around that". The advantage of being on totally clean boxes I thought was "elastic can call the shots, everything else will need to fit around that". I do take on board the difference between elastic cluster and the /etc/passwd users. I just wasn't sure (and did not see it in any documentation that I have look at), as to what I should be logged in with to do the install. I have to be logged in as _something_ to install elasticsearch - and just wondered if this should be root, \ **Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [May 28, 2025, 11:45am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/4 "2025-05-28T11:45:04Z") Modern Linux distros discourage direct use of the root user, and usually at OS installation time create a user with admin (via sudo) privileges. Old school like me might not be a massive fan of sudo, but I've given up and just accept thats the way it is now. When you come to install Elasticsearch from a yum repo, you will see output like: ```auto Running transaction Preparing : 1/1 Running scriptlet: elasticsearch-9.0.1-1.x86_64 1/1 Creating elasticsearch group... OK Creating elasticsearch user... OK ``` where it creates the relevant users who will own the elasticsearch processes. ```auto [kevin@rhel9x ~]$ id elasticsearch uid=979(elasticsearch) gid=978(elasticsearch) groups=978(elasticsearch) ``` In RHEL case you will also see various output like: ```auto --------------------------- Security autoconfiguration information ------------------------------ Authentication and authorization are enabled. TLS for the transport and HTTP layers is enabled and configured. The generated password for the elastic built-in superuser is : whateveritischosenrandomly ... ``` As it happens my RHEL 9 has this version of openjdk installed already, probably by some by some dependency. ```auto [kevin@rhel9x ~]$ java -version openjdk version "1.8.0_442" OpenJDK Runtime Environment (build 1.8.0_442-b06) OpenJDK 64-Bit Server VM (build 25.442-b06, mixed mode) ``` But elasticsearch ships with and uses it's own by default, and in your shoes I would not change this. ```auto [kevin@rhel9x ~]$ /usr/share/elasticsearch/jdk/bin/java -version openjdk version "24" 2025-03-18 OpenJDK Runtime Environment (build 24+36-3646) OpenJDK 64-Bit Server VM (build 24+36-3646, mixed mode, sharing) ``` I like using VMs to learn. Do a base OS install, document what you did, make a snapshot. Then install say elasticsearch and associated needed tools, document your setup process, making notes, until you are happy. Then revert everything to your base snapshot and see if your notes are good enough to go from zero to hero. if so, you are done. If not, improve your notes. Iterate til done. Use that process to instal for real on the real hardware. ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [May 28, 2025, 12:05pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/5 "2025-05-28T12:05:55Z")
Fantastic feedback, thank you. I have installed VMs on a little linux laptop I have, but had not installed it on a project servers. I think that is my next little task! Makes excellent sense, and I will start to work with that today. The snapshot idea will make "starting over" a "while I pop the kettle on" job instead of a 1 hour job. Originally I just thought "utterly clean install, nothing else loaded - should install great!".... (I can picture you laughing πŸ™‚ ) Again, many thanks for offering simple, practical guidance, it really helps! ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [June 16, 2025, 7:22pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/6 "2025-06-16T19:22:50Z")
Update: This is crazy but I am struggling now to even install VMWARE. All of the guides I can find no longer work since Broadcom took over BUT I have managed to download it. My issue is that it wants to compile various things before it can run - and although I have done the updates and updated the headers, it now complains because although I have the LATEST headers, it is seeking OLDER ones - and I am not sure how to either download old headers for it to play with OR make it use new headers. I would understand lots of teething issues with botched installs on heavily modified and tweeked systems - but this much drama to install either Elasticsearch or VMWARE on utterly clean RHEL installations does seem daft. I don't seem to recall this much "fun" 40 years ago with Unix and Xenix. You extracted, ran, it installed and everything just seemed to work. Anyway, is there a simple way to either tell VMWARE to use the LATEST headers OR for me to be able to download the OLD ones for it please? I mean I don't need VMWARE other than to try to find different ways to make a completely clean installation of Elastic worth on a completely clean RHEL server - but it did seem as though this might allow me to easily start over each time ES failed again. Is there an issue with RHEL where if I simply switch to something like Rocky Linux (or any other version), a clean install of ES on a clean install of linux is not likely to end up with security issue about corrupted keystores - or complain about security being set when I have specifically switched off ALL security in the yml. I realise it must be me doing something wrong, but using Google just finds a lot of others with remarkably similar issues - but with no posted resolutions! Bring back Dos 5 (or 6.22) πŸ™‚ ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [June 16, 2025, 9:36pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/7 "2025-06-16T21:36:56Z")
I am a little lost. I think you are over-complicating things maybe based on reading too much. To install single-node elasticsearch instance onto a fresh RHEL install should be 1) import a key, 2) add a yum repo, 3) install the package - yum/dnf does all the work. To make a cluster would require a _little_ more work, but not much. The subject mentions "persistent keystore errors", and you mentioned again "corrupted keystones", but if I've understood correctly these were problems _last time_, not currently? If thats not the case, please share/show more information incl which errors you are getting now? As for installing VMware, well it's a while since I used it but I guess it needs build kernel modules. I use VirtualBox, but there's not much difference really, could use virt-install/virt-manager too. There's really too many options for virtualisation nowadays πŸ˜‰ I just so happened to install a RHEL 10 (test) system yesterday, let give it a go, not officially supported yet but ... will be much the same as on RHEL 8/9. I will follow: > **[Install Elasticsearch with RPM | Elastic Docs](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm)** > > The RPM package for Elasticsearch can be downloaded from our website or from our RPM repository. It can be used to install Elasticsearch on any RPM-based... Note it seemed to work, with only a minor whine around `var/run` which is now a link to `/run` ```auto /usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch β†’ /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly. ``` which is not important for here. Here's the full process, my user (kevin) has sudo rights. ```auto [kevin@rhel10x1 ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch [kevin@rhel10x1 ~]$ sudo vi /etc/yum.repos.d/elastic.repo < pasted the contents shown below > [kevin@rhel10x1 ~]$ sudo cat /etc/yum.repos.d/elastic.repo [elasticsearch] name=Elasticsearch repository for 9.x packages baseurl=https://artifacts.elastic.co/packages/9.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=0 autorefresh=1 type=rpm-md [kevin@rhel10x1 ~]$ id elasticsearch id: β€˜elasticsearch’: no such user [kevin@rhel10x1 ~]$ sudo dnf install --enablerepo=elasticsearch elasticsearch -y Updating Subscription Management repositories. Last metadata expiration check: 0:11:58 ago on Mon Jun 16 21:06:05 2025. Dependencies resolved. ======================================================================================================================== Package Architecture Version Repository Size ======================================================================================================================== Installing: elasticsearch x86_64 9.0.2-1 elasticsearch 620 M Transaction Summary ======================================================================================================================== Install 1 Package Total download size: 620 M Installed size: 1.2 G Downloading Packages: elasticsearch-9.0.2-x86_64.rpm 5.5 MB/s | 620 MB 01:52 ------------------------------------------------------------------------------------------------------------------------ Total 5.5 MB/s | 620 MB 01:52 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: elasticsearch-9.0.2-1.x86_64 1/1 Creating elasticsearch group... OK Creating elasticsearch user... OK Installing : elasticsearch-9.0.2-1.x86_64 1/1 Running scriptlet: elasticsearch-9.0.2-1.x86_64 1/1 --------------------------- Security autoconfiguration information ------------------------------ Authentication and authorization are enabled. TLS for the transport and HTTP layers is enabled and configured. The generated password for the elastic built-in superuser is : UNMkFuZcZNH=QRg9q22F If this node should join an existing cluster, you can reconfigure this with '/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token ' after creating an enrollment token on your existing cluster. You can complete the following actions at any time: Reset the password of the elastic built-in superuser with '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'. Generate an enrollment token for Kibana instances with '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'. Generate an enrollment token for Elasticsearch nodes with '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'. ------------------------------------------------------------------------------------------------- ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch β†’ /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly. Installed products updated. Installed: elasticsearch-9.0.2-1.x86_64 Complete! [kevin@rhel10x1 ~]$ sudo systemctl start elasticsearch.service [kevin@rhel10x1 ~]$ sudo systemctl status elasticsearch.service | cat ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; preset: disabled) Active: active (running) since Mon 2025-06-16 21:22:43 UTC; 24s ago Invocation: ab33179b9abc44ecafcb9f0a019d8d62 Docs: https://www.elastic.co Main PID: 5205 (java) Tasks: 111 (limit: 10395) Memory: 1.3G (peak: 1.3G) CPU: 40.407s CGroup: /system.slice/elasticsearch.service β”œβ”€5205 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.type=rpm "-Des.java.type=bundled JDK" -cp "/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/cli-launcher/*" org.elasticsearch.launcher.CliToolLauncher -p /var/run/elasticsearch/elasticsearch.pid --quiet β”œβ”€5267 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=CLDR -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24 -Des.distribution.type=rpm "-Des.java.type=bundled JDK" --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core --enable-native-access=ALL-UNNAMED --illegal-native-access=deny -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log -Des.entitlements.enabled=true -XX:+EnableDynamicAgentLoading -Djdk.attach.allowAttachSelf=true --patch-module=java.base=lib/entitlement-bridge/elasticsearch-entitlement-bridge-9.0.2.jar --add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-7540940937904985010 --add-modules=jdk.incubator.vector -Dorg.apache.lucene.store.defaultReadAdvice=normal -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m" -Xms852m -Xmx852m -XX:MaxDirectMemorySize=446693376 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path /usr/share/elasticsearch/lib --add-modules=jdk.net --add-modules=jdk.management.agent --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch └─5287 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller Jun 16 21:21:56 rhel10x1 systemd[1]: Starting elasticsearch.service - Elasticsearch... Jun 16 21:22:43 rhel10x1 systemd[1]: Started elasticsearch.service - Elasticsearch. [kevin@rhel10x1 ~]$ EUSER=elastic EPASS="UNMkFuZcZNH=QRg9q22F" [kevin@rhel10x1 ~]$ curl -sku "${EUSER}:${EPASS}" https://localhost:9200 | jq -r . { "name": "rhel10x1", "cluster_name": "elasticsearch", "cluster_uuid": "_wrTbkCRQauLMU8ZfYXtZA", "version": { "number": "9.0.2", "build_flavor": "default", "build_type": "rpm", "build_hash": "0a58bc1dc7a4ae5412db66624aab968370bd44ce", "build_date": "2025-05-28T10:06:37.834829258Z", "build_snapshot": false, "lucene_version": "10.1.0", "minimum_wire_compatibility_version": "8.18.0", "minimum_index_compatibility_version": "8.0.0" }, "tagline": "You Know, for Search" } [kevin@rhel10x1 ~]$ curl -sku "${EUSER}:${EPASS}" https://localhost:9200/_cat/indices [kevin@rhel10x1 ~]$ curl -sku "${EUSER}:${EPASS}" "https://localhost:9200/_cat/indices?index=.*" green open .security-7 h3oQBa82SWqOwGEayi3T_g 1 0 30 0 40.9kb 40.9kb 40.9kb green open .ds-ilm-history-7-2025.06.16-000001 OcN-xcEiRniz30GpGO5Ouw 1 0 3 0 9.7kb 9.7kb 9.7kb green open .ds-.logs-elasticsearch.deprecation-default-2025.06.16-000001 497jow5-S_qExb7LJzjNxA 1 0 1 0 10.2kb 10.2kb 10.2kb [kevin@rhel10x1 ~]$ cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="10.0 (Coughlan)" ID="rhel" ID_LIKE="centos fedora" VERSION_ID="10.0" PLATFORM_ID="platform:el10" PRETTY_NAME="Red Hat Enterprise Linux 10.0 (Coughlan)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:10::baseos" HOME_URL="https://www.redhat.com/" VENDOR_NAME="Red Hat" VENDOR_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10" BUG_REPORT_URL="https://issues.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 10" REDHAT_BUGZILLA_PRODUCT_VERSION=10.0 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="10.0" [kevin@rhel10x1 ~]$ head -1 /proc/meminfo MemTotal: 1746136 kB [kevin@rhel10x1 ~]$ egrep ^proc /proc/cpuinfo processor : 0 processor : 1 [kevin@rhel10x1 ~]$ [kevin@rhel10x1 ~]$ sudo ls -ld /var/run /run drwxr-xr-x. 38 root root 1140 Jun 16 21:21 /run lrwxrwxrwx. 1 root root 6 Apr 23 14:35 /var/run -> ../run [kevin@rhel10x1 ~]$ id elasticsearch uid=990(elasticsearch) gid=990(elasticsearch) groups=990(elasticsearch) [kevin@rhel10x1 ~]$ ps -uelasticsearch -opid,command | cat PID COMMAND 5205 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.type=rpm -Des.java.type=bundled JDK -cp /usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher -p /var/run/elasticsearch/elasticsearch.pid --quiet 5267 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=CLDR -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24 -Des.distribution.type=rpm -Des.java.type=bundled JDK --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core --enable-native-access=ALL-UNNAMED --illegal-native-access=deny -XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log -Des.entitlements.enabled=true -XX:+EnableDynamicAgentLoading -Djdk.attach.allowAttachSelf=true --patch-module=java.base=lib/entitlement-bridge/elasticsearch-entitlement-bridge-9.0.2.jar --add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-7540940937904985010 --add-modules=jdk.incubator.vector -Dorg.apache.lucene.store.defaultReadAdvice=normal -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Xms852m -Xmx852m -XX:MaxDirectMemorySize=446693376 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path /usr/share/elasticsearch/lib --add-modules=jdk.net --add-modules=jdk.management.agent --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch 5287 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller ``` ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [June 18, 2025, 8:13pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/8 "2025-06-18T20:13:37Z")
Thank you. I will try this. Possibly the mistake I have made is that when I explained to folks I wanted to setup a cluster, it was suggested to edit the yml files giving details of master intitial nodes etc - and also the instances file so each node knows the IP addresses of the others. As such I created custom yml files (example of one below) so I could generate the security certs in one go - and then copy them to the other nodes. I am very happy to just create a single node - which I hope to be able to get working tonight - but then, I am just concerned that making it "make friends with other nodes" will then be more difficult. Anyway, I will have a go installing EXACTLY as you suggest - which I suspect will work, (and Thank you either way for your time and kindness). My concern will then be how to make them "find each other", but I guess first step first πŸ™‚ Again, thank you for your time, I shall do a completely clean install now with RHEL 9 and try with your guide. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [June 18, 2025, 8:48pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/9 "2025-06-18T20:48:06Z")
> [@GuyMark](#): > > As such I created custom yml files (example of one below) I guess you forgot to include the yml file. Good luck with the RHEL/elasticsearch installation. [Just FYI, I can tell you for 90%+ of the queries / questions I try to answer on here, I use a single elasticsearch and single kibana instance (both running on my M1 Mac Mini) to check stuff. It also houses some actual data and an irregularly-taken (do as I say, not as I do!) snapshot repo, the repo backed up via Time Machine (I really should use another USB drive for a more resilient home for my repo). I have a SFF miniPC (cost me 250 euro, I'm Scottish!) system hanging off the back of my monitor (literally) on which I have Ubuntu, and run VirtualBox, docker, podman, minikube, ... eg the RHEL10 was a quick install using virt-install. Moral of all of this is nowadays you can do a lot, and learn a lot, with relatively little.] ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [June 18, 2025, 10:05pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/10 "2025-06-18T22:05:09Z")
Apologies, you are quite right, I ended up copying the yml into my notes instead of on here. This is what I was using - though there were other iterations tried.:- ```auto cluster.name: testcluster node.name: es1 network.host: 172.22.22.131 discovery.seed_hosts: ["172.22.22.131", "172.22.22.132", "172.22.22.133"] cluster.initial_master_nodes: ["es1", "es2", "es3"] node.roles: [master, data, remote_cluster_client] xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/es1.p12 xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/es1.p12 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch ``` For now though, I have not touched the elasticsearch.yml or instances.yml Your comments on using VM for doing various test is excellent - and I have managed to install it on a Mint Cinnamon laptop - but struggling to install vmware on an HP DL360 G9. I will respond when I have ES installed and stable! ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [June 19, 2025, 12:09am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/11 "2025-06-19T00:09:22Z")
A short update before bed. Thank you so much for your patient assistance, I now have a working single node, I will make another on Friday and see if I can make them "see" each other. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [June 19, 2025, 1:47am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/12 "2025-06-19T01:47:38Z")
Thanks for update. I suggest also to install lsof on all 3 nodes: `sudo dnf install lsof -y` and then `sudo lost -i :9200` and `ps -uelasticsearch -opid= | xargs -n1 sudo lsof -nPp | fgrep TCP` on the single node that is installed/working. And you should have close to the "vendor default" elasticsearch,yml on that node too `sudo cat /etc/elasticsearch/elasticsearch.yml | grep -Ev \^# | egrep . | cat` The gotchas you might need to avoid are elasticsearch only listening on loopback interfaces by default, the `lsof` commands can show you this, and messing about with discovery and initial-nodes settings that some guides suggest. Assuming you have your enrollment token from the output when node1 was setup, and you can always regenerate one anyways, on **other** nodes it should be as simple as `sudo /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token aVeryLongToken==` You can use same token for all other nodes. IMHO once you have all the nodes in the cluster, you should tidy up/align the elasticsearch,yml files, my own 3-node cluster has ended up with following settings: ```auto [kevin@rhel10x3 ~]$ sudo cat /etc/elasticsearch/elasticsearch.yml | grep -Ev \^# | egrep . | cat path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 discovery.seed_hosts: ["rhel10x1:9300", "rhel10x2:9300","rhel10x3:9300"] http.host: 0.0.0.0 transport.host: _site_ ``` (I had already added the hostname entries in /etc/hosts on all nodes to avoid remembering the local IP addresses) ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [June 22, 2025, 1:45pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/13 "2025-06-22T13:45:19Z")
> [@GuyMark](#): > > I will respond when I have ES installed and stable! > [@GuyMark](#): > > I now have a working single node How did you get on with the next steps? ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [June 23, 2025, 12:56am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/14 "2025-06-23T00:56:19Z")
Hello. I now have three isolated stand-alone nodes which all load up independently - which I would like to work together as the initial masters and the "always on set". These boxes will not run anything other than Elasticsearch (other than updates / etc) I also have one further node which is on a shared box. It will often be switched off, but when on, it may need to be able to join the cluster, as its job is to prepare data and then ingest it. I am trying to do as much as I can without pestering folks like yourself - so working out at the moment things like "How to tell elasticsearch that it needs to use network port #2 to reach the rest of the cluster" and so on. Once I have done that though, I think I am going to need a little guidance as I suspect once I start editing the yml files, I may end up with some more "fun". So far so good though - as I have four boxes which all seem to be happy to run Elasticsearch - even if, at the moment, they do not know the other nodes exist. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [June 23, 2025, 6:51am UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/15 "2025-06-23T06:51:24Z")
Thanks for update. You seem on right trajectory. > [@GuyMark](#): > > I also have one further node which is on a shared box. It will often be switched off, but when on, it may need to be able to join the cluster, as its job is to prepare data and then ingest it. I simply would not include/add this to the cluster, based on what you wrote there. Preparing and ingesting data can be done from outside the cluster footprint. Others might disagree on this. If your eventual cluster yml file has more than a dozen settings/lines, it’s likely bloated. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 2, 2025, 9:14pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/16 "2025-07-02T21:14:39Z")
@GuyMark did you reach Nirvana ? ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [July 2, 2025, 11:09pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/17 "2025-07-02T23:09:45Z")
Not yet! But I have managed to get pretty much all of Friday set aside to have another go at making them talk to each other. I noted what you said about not using ingestion nodes to ingest data and I have not managed to find much info on that - but as I have not yet managed to get the initial three nodes happily talking, I have not spent too much time on that yet. If I cannot sort on Friday, I will give specific issues that I am struggling with on the basis that it is likely to be something daft. ---
**Author:** ![GuyMark](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@GuyMark](https://discuss.elastic.co/u/GuyMark)\ **Post date:** [July 18, 2025, 11:09pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/18 "2025-07-18T23:09:03Z")
Some progress πŸ™‚ I was unduly concerned about some error messages which are apparently ignorable when running:- $ ps -uelasticsearch -opid= | xargs -n1 sudo lsof -nPp | fgrep TCP I am now using RHEL 10 and that seems to have lsof installed by default. However when I try the command:- $ sudo /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token mytoken I keep getting the following. ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only., with exit code 80 Other than making sure it installed, nodes 2 and 3 have never been started before, nor do they have any data on them. I have re-installed nodes 2 and 3 a few times but cannot find a way to override this error. I have tried Google - but following that was what made things "more fun" in the first place. I have not yet edited the yml file, as you suggested to do that NEXT. So I am figuring the defaults will help it find each other (all servers have fixed IP and are connected to a switch that has nothing else connected other than PORT 4 of each box). If you think I should edit the yml file FIRST before trying again, will happily do so, but thought I would query first. Thank you for your support so far, it has been extremely helpful. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 19, 2025, 5:18pm UTC](https://discuss.elastic.co/t/persistent-keystore-errors-guidance-for-new-install-please/378620/19 "2025-07-19T17:18:12Z")
Yeah, it's not clear to me where you are with nodes 2/3. You might think elasticsearch was never started on the nodes, but ... might not be true. What do you get for this command on all nodes? ```auto sudo ls /var/lib/elasticsearch/ ``` I get ```auto $ sudo ls /var/lib/elasticsearch/ _state indices node.lock nodes snapshot_cache ``` because my (3-node) cluster has been running for a while. The ps command is just too heck the port is open (on 1st node), this variant is easier and should return something similar on your system ```auto [kevin@rhel10x1 ~]$ sudo lsof -Pni :9200 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 1658 elasticsearch 572u IPv6 12423 0t0 TCP *:9200 (LISTEN) ``` Important is that its listening on a non-loopback interface. > [@GuyMark](#): > > I have not yet edited the yml file well, on nodes 2/3 thats fine. But whats on node1? Mine (now that I have 3-node cluster) is: ```auto [kevin@rhel10x1 ~]$ sudo grep -E '^[^#].*' /etc/elasticsearch/elasticsearch.yml path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch discovery.seed_hosts: ["rhel10x1:9300", "rhel10x2:9300","rhel10x3:9300"] xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0 transport.host: _site_ xpack.monitoring.elasticsearch.collection.enabled: true xpack.monitoring.collection.enabled: true ``` yours would not have the same settings for `discovery.seed_hosts`, but otherwise should be pretty similar.