Failing to Start - License issue?

I’m trying to start ES 9.0.2 as a service. Every time, the service fails with an error 143. One thing of note is that in the logs its telling me my license is not valid despite just using a basic license. I can start elasticsearch just fine using elasticsearch -d -p pid but not from a systemd unit file.

[Unit]
2 Description=Elasticsearch
3 Documentation=https://www.elastic.co
4 Wants=network-online.target
5 After=network-online.target
6
7 [Service]
8 Type=notify
9 # the elasticsearch process currently sends the notifications back to systemd
10 # and for some reason exec does not work (even though it is a child). We should change
11 # this notify access back to main (the default), see https://github.com/elastic/elasticsearch/issues/86475
12 NotifyAccess=all
13 RuntimeDirectory=elasticsearch
14 PrivateTmp=true
15 Environment=ES_HOME=/opt/elasticsearch
16 Environment=ES_PATH_CONF=/opt/elasticsearch/config
17 Environment=PID_DIR=/var/run/elasticsearch
18 Environment=ES_SD_NOTIFY=true
19 EnvironmentFile=/opt/elasticsearch/config/elasticsearch.yml
20
21 WorkingDirectory=/opt/elasticsearch
22
23 User=elasticsearch
24 Group=elasticsearch
25
26 ExecStart=/opt/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid
27
[2026-01-11T02:17:11,452][INFO ][o.e.b.Elasticsearch      ] [MY_DOMAIN] version[9.0.2], pid[2566146], build[tar/0a58bc1dc7a4ae5412db66624aab968370bd44ce/2025-05-28T10:06:37.834829258Z], OS[Linux/6.8.0-60-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/24/24+36-3646]
[2026-01-11T02:17:11,471][INFO ][o.e.b.Elasticsearch      ] [MY_DOMAIN] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
[2026-01-11T02:17:11,472][INFO ][o.e.b.Elasticsearch      ] [MY_DOMAIN] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=CLDR, -Dorg.apache.lucene.vectorization.upperJavaFeatureVersion=24, -Des.distribution.type=tar, -Des.java.type=bundled JDK, --enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core, --enable-native-access=ALL-UNNAMED, --illegal-native-access=deny, -XX:ReplayDataFile=logs/replay_pid%p.log, -Des.entitlements.enabled=true, -XX:+EnableDynamicAgentLoading, -Djdk.attach.allowAttachSelf=true, --patch-module=java.base=lib/entitlement-bridge/elasticsearch-entitlement-bridge-9.0.2.jar, --add-exports=java.base/org.elasticsearch.entitlement.bridge=org.elasticsearch.entitlement,java.logging,java.net.http,java.naming,jdk.net, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9052815450537628127, --add-modules=jdk.incubator.vector, -Dorg.apache.lucene.store.defaultReadAdvice=normal, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, --module-path=/opt/elasticsearch/lib, --add-modules=jdk.net, --add-modules=jdk.management.agent, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2026-01-11T02:17:11,473][INFO ][o.e.b.Elasticsearch      ] [MY_DOMAIN] Default Locale [en_US]
[2026-01-11T02:17:11,602][INFO ][o.e.n.j.JdkVectorLibrary ] [MY_DOMAIN] vec_caps=1
[2026-01-11T02:17:11,606][INFO ][o.e.n.NativeAccess       ] [MY_DOMAIN] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2026-01-11T02:17:11,617][INFO ][o.e.n.NativeAccess       ] [MY_DOMAIN] Using [jdk] native provider and native methods for [Linux]
[2026-01-11T02:17:11,674][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [MY_DOMAIN] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2026-01-11T02:17:11,703][INFO ][o.e.b.Elasticsearch      ] [MY_DOMAIN] Bootstrapping Entitlements
[2026-01-11T02:17:14,541][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [repository-url]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [rest-root]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-core]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-redact]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [ingest-user-agent]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-async-search]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-monitoring]
[2026-01-11T02:17:14,542][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [repository-s3]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-analytics]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-esql-core]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-ent-search]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-autoscaling]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [lang-painless]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-ml]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [legacy-geo]
[2026-01-11T02:17:14,543][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [lang-mustache]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [logsdb]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-ql]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [rank-rrf]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [analysis-common]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [health-shards-availability]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [transport-netty4]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [aggregations]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [ingest-common]
[2026-01-11T02:17:14,544][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-identity-provider]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [frozen-indices]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-shutdown]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-text-structure]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [snapshot-repo-test-kit]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [ml-package-loader]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [kibana]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [constant-keyword]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-logstash]
[2026-01-11T02:17:14,545][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-graph]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-ccr]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [rank-vectors]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-esql]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [parent-join]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [counted-keyword]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-enrich]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [repositories-metering-api]
[2026-01-11T02:17:14,546][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [transform]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [repository-azure]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [dot-prefix-validation]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [repository-gcs]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [spatial]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-otel-data]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [mapper-extras]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [apm]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [mapper-version]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-rollup]
[2026-01-11T02:17:14,547][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [percolator]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-migrate]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-stack]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [data-streams]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [reindex]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [rank-eval]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-security]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [blob-cache]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [searchable-snapshots]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-slm]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-geoip-enterprise-downloader]
[2026-01-11T02:17:14,548][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [snapshot-based-recoveries]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-watcher]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [old-lucene-versions]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-ilm]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-voting-only-node]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-inference]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-deprecation]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-fleet]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-aggregate-metric]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-downsample]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-profiling]
[2026-01-11T02:17:14,549][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [ingest-geoip]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-write-load-forecaster]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [search-business-rules]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [wildcard]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [ingest-attachment]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-apm-data]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [unsigned-long]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-sql]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-async]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [runtime-fields-common]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [vector-tile]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-kql]
[2026-01-11T02:17:14,550][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [lang-expression]
[2026-01-11T02:17:14,551][INFO ][o.e.p.PluginsService     ] [MY_DOMAIN] loaded module [x-pack-eql]
[2026-01-11T02:17:15,575][INFO ][o.e.e.NodeEnvironment    ] [MY_DOMAIN] using [1] data paths, mounts [[/ (/dev/nvme0n1p3)]], net usable_space [2.7tb], net total_space [6.9tb], types [ext4]
[2026-01-11T02:17:15,575][INFO ][o.e.e.NodeEnvironment    ] [MY_DOMAIN] heap size [31gb], compressed ordinary object pointers [true]
[2026-01-11T02:17:15,614][INFO ][o.e.n.Node               ] [MY_DOMAIN] node name [MY_DOMAIN], node ID [fCV3jz9DTWGeC2iveLwB6A], cluster name [MY_DOMAIN], roles [data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot, transform]
[2026-01-11T02:17:17,378][INFO ][o.e.i.r.RecoverySettings ] [MY_DOMAIN] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2026-01-11T02:17:17,478][INFO ][o.e.f.FeatureService     ] [MY_DOMAIN] Registered local node features [ES_V_8, ES_V_9, cluster.reroute.ignores_metric_param, cluster.stats.source_modes, linear_retriever_supported, lucene_10_1_upgrade, lucene_10_upgrade, security.queryable_built_in_roles, simulate.ignored.fields]
[2026-01-11T02:17:17,502][INFO ][o.e.c.m.DataStreamGlobalRetentionSettings] [MY_DOMAIN] Updated default factory retention to [null]
[2026-01-11T02:17:17,503][INFO ][o.e.c.m.DataStreamGlobalRetentionSettings] [MY_DOMAIN] Updated max factory retention to [null]
[2026-01-11T02:17:17,748][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [MY_DOMAIN] [controller/2566265] [Main.cc@123] controller (64 bit): Version 9.0.2 (Build b88ad275a00fe7) Copyright (c) 2025 Elasticsearch BV
[2026-01-11T02:17:17,988][INFO ][o.e.x.o.OTelPlugin       ] [MY_DOMAIN] OTel ingest plugin is enabled
[2026-01-11T02:17:18,008][INFO ][o.e.x.c.t.YamlTemplateRegistry] [MY_DOMAIN] OpenTelemetry index template registry is enabled
[2026-01-11T02:17:18,011][INFO ][o.e.t.a.APM              ] [MY_DOMAIN] Sending apm metrics is disabled
[2026-01-11T02:17:18,011][INFO ][o.e.t.a.APM              ] [MY_DOMAIN] Sending apm tracing is disabled
[2026-01-11T02:17:18,033][INFO ][o.e.x.s.Security         ] [MY_DOMAIN] Security is enabled
[2026-01-11T02:17:18,296][INFO ][o.e.x.s.a.s.FileRolesStore] [MY_DOMAIN] parsed [0] roles from file [/opt/elasticsearch/config/roles.yml]
[2026-01-11T02:17:18,427][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [MY_DOMAIN] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot  determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2026-01-11T02:17:18,605][INFO ][o.e.x.w.Watcher          ] [MY_DOMAIN] Watcher initialized components at 2026-01-11T08:17:18.605Z
[2026-01-11T02:17:18,672][INFO ][o.e.x.p.ProfilingPlugin  ] [MY_DOMAIN] Profiling is enabled
[2026-01-11T02:17:18,683][INFO ][o.e.x.p.ProfilingPlugin  ] [MY_DOMAIN] profiling index templates will not be installed or reinstalled
[2026-01-11T02:17:18,686][INFO ][o.e.x.a.APMPlugin        ] [MY_DOMAIN] APM ingest plugin is enabled
[2026-01-11T02:17:18,708][INFO ][o.e.x.c.t.YamlTemplateRegistry] [MY_DOMAIN] apm index template registry is enabled
[2026-01-11T02:17:19,004][INFO ][o.e.t.n.NettyAllocator   ] [MY_DOMAIN] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2026-01-11T02:17:19,040][INFO ][o.e.d.DiscoveryModule    ] [MY_DOMAIN] using discovery type [single-node] and seed hosts providers [settings]
[2026-01-11T02:17:19,834][INFO ][o.e.n.Node               ] [MY_DOMAIN] initialized
[2026-01-11T02:17:19,834][INFO ][o.e.n.Node               ] [MY_DOMAIN] starting ...
[2026-01-11T02:17:19,844][INFO ][o.e.x.s.c.f.PersistentCache] [MY_DOMAIN] persistent cache index loaded
[2026-01-11T02:17:19,845][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [MY_DOMAIN] deprecation component started
[2026-01-11T02:17:19,892][INFO ][o.e.t.TransportService   ] [MY_DOMAIN] publish_address {INTERNAL_IP_ADDR:9300}, bound_addresses {INTERNAL_IP_ADDR:9300}
[2026-01-11T02:17:20,074][INFO ][o.e.c.c.ClusterBootstrapService] [MY_DOMAIN] this node is locked into cluster UUID [yAs67_OPR_-CRDauU8BbhQ] and will not attempt further cluster bootstrapping
[2026-01-11T02:17:20,161][INFO ][o.e.c.s.MasterService    ] [MY_DOMAIN] elected-as-master ([1] nodes joined in term 22)[_FINISH_ELECTION_, {MY_DOMAIN}{fCV3jz9DTWGeC2iveLwB6A}{ZUVSP2bGQbCrQFT-xYF_vA}{MY_DOMAIN}{INTERNAL_IP_ADDR}{INTERNAL_IP_ADDR:9300}{cdfhilmrstw}{9.0.2}{8000099-9009000} completing election], term: 22, version: 345, delta: master node changed {previous [], current [{MY_DOMAIN}{fCV3jz9DTWGeC2iveLwB6A}{ZUVSP2bGQbCrQFT-xYF_vA}{MY_DOMAIN}{INTERNAL_IP_ADDR}{INTERNAL_IP_ADDR:9300}{cdfhilmrstw}{9.0.2}{8000099-9009000}]}
[2026-01-11T02:17:20,192][INFO ][o.e.c.s.ClusterApplierService] [MY_DOMAIN] master node changed {previous [], current [{MY_DOMAIN}{fCV3jz9DTWGeC2iveLwB6A}{ZUVSP2bGQbCrQFT-xYF_vA}{MY_DOMAIN}{INTERNAL_IP_ADDR}{INTERNAL_IP_ADDR:9300}{cdfhilmrstw}{9.0.2}{8000099-9009000}]}, term: 22, version: 345, reason: Publication{term=22, version=345}
[2026-01-11T02:17:20,228][INFO ][o.e.c.c.NodeJoinExecutor ] [MY_DOMAIN] node-join: [{MY_DOMAIN}{fCV3jz9DTWGeC2iveLwB6A}{ZUVSP2bGQbCrQFT-xYF_vA}{MY_DOMAIN}{INTERNAL_IP_ADDR}{INTERNAL_IP_ADDR:9300}{cdfhilmrstw}{9.0.2}{8000099-9009000}] with reason [completing election]
[2026-01-11T02:17:20,229][INFO ][o.e.h.AbstractHttpServerTransport] [MY_DOMAIN] publish_address {INTERNAL_IP_ADDR:9200}, bound_addresses {INTERNAL_IP_ADDR:9200}
[2026-01-11T02:17:20,235][INFO ][o.e.x.w.LicensedWriteLoadForecaster] [MY_DOMAIN] license state changed, now [valid]
[2026-01-11T02:17:20,236][INFO ][o.e.n.Node               ] [MY_DOMAIN] started {MY_DOMAIN}{fCV3jz9DTWGeC2iveLwB6A}{ZUVSP2bGQbCrQFT-xYF_vA}{MY_DOMAIN}{INTERNAL_IP_ADDR}{INTERNAL_IP_ADDR:9300}{cdfhilmrstw}{9.0.2}{8000099-9009000}{xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=134985830400, ml.allocated_processors=24, ml.allocated_processors_double=24.0, ml.max_jvm_size=33285996544, ml.config_version=12.0.0}
[2026-01-11T02:17:20,266][WARN ][o.e.x.i.s.e.a.ElasticInferenceServiceAuthorizationHandler] [MY_DOMAIN] Failed to revoke access to default inference endpoint IDs: [rainbow-sprinkles], error: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
[2026-01-11T02:17:20,492][INFO ][o.e.x.m.MlIndexRollover  ] [MY_DOMAIN] ML legacy indices rolled over
[2026-01-11T02:17:20,492][INFO ][o.e.x.m.MlAnomaliesIndexUpdate] [MY_DOMAIN] legacy ml anomalies indices rolled over and aliases updated
[2026-01-11T02:17:20,493][INFO ][o.e.x.s.a.Realms         ] [MY_DOMAIN] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2026-01-11T02:17:20,495][INFO ][o.e.l.ClusterStateLicenseService] [MY_DOMAIN] license [MY_LICENSE] mode [basic] - valid
[2026-01-11T02:17:20,496][INFO ][o.e.c.f.AbstractFileWatchingService] [MY_DOMAIN] starting file watcher ...
[2026-01-11T02:17:20,499][INFO ][o.e.c.f.AbstractFileWatchingService] [MY_DOMAIN] file settings service up and running [tid=92]
[2026-01-11T02:17:20,500][INFO ][o.e.r.s.FileSettingsService] [MY_DOMAIN] setting file [/opt/elasticsearch/config/operator/settings.json] not found, initializing [file_settings] as empty
[2026-01-11T02:17:20,507][INFO ][o.e.g.GatewayService     ] [MY_DOMAIN] recovered [4] indices into cluster_state
[2026-01-11T02:17:20,540][INFO ][o.e.x.w.LicensedWriteLoadForecaster] [MY_DOMAIN] license state changed, now [not valid]
[2026-01-11T02:17:20,722][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [MY_DOMAIN] Node [{MY_DOMAIN}{fCV3jz9DTWGeC2iveLwB6A}] is selected as the current health node.
[2026-01-11T02:17:20,744][INFO ][o.e.c.r.a.AllocationService] [MY_DOMAIN] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[MY_INDEX][0], [.security-7][0]]])." previous.health="RED" reason="shards started [[MY_INDEX][0], [.security-7][0]]"

Hello @aurora

Can you please share the logs where it shows that elasticsearch did not start & or it failed?

One similar post related to this issue is below if it can be helpful :

Thanks!!

This is all I have to work off of :\

× elasticsearch.service - Elasticsearch
Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; preset: enabled)
Active: failed (Result: timeout) since Sun 2026-01-11 02:32:12 CST; 8h ago
Docs: https://www.elastic.co
Process: 2566030 ExecStart=/opt/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid (code=exited, status=143)
Main PID: 2566030 (code=exited, status=143)
CPU: 1min 37.553s

Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,298][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [MY_DOMAIN] [controller/2566265] [Main.cc@176] ML controller exiting
Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,298][INFO ][o.e.x.w.t.s.e.TickerScheduleTriggerEngine] [MY_DOMAIN] Stopping watcher engine
Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,298][INFO ][o.e.x.w.WatcherLifeCycleService] [MY_DOMAIN] watcher has stopped and shutdown
Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,298][INFO ][o.e.x.m.p.NativeController] [MY_DOMAIN] Native controller process has stopped - no new native processes can be started
Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,790][INFO ][o.e.n.Node               ] [MY_DOMAIN] stopped
Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,790][INFO ][o.e.n.Node               ] [MY_DOMAIN] closing ...
Jan 11 02:32:07 MY_DOMAIN elasticsearch[2566146]: [2026-01-11T02:32:07,801][INFO ][o.e.n.Node               ] [MY_DOMAIN] closed
Jan 11 02:32:12 MY_DOMAIN systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Jan 11 02:32:12 MY_DOMAIN systemd[1]: Failed to start elasticsearch.service - Elasticsearch.
Jan 11 02:32:12 MY_DOMAIN systemd[1]: elasticsearch.service: Consumed 1min 37.553s CPU time.

[2026-01-11T02:32:07,295][INFO ][o.e.n.Node               ] [MY_DOMAIN] stopping ...
144 [2026-01-11T02:32:07,296][INFO ][o.e.c.f.AbstractFileWatchingService] [MY_DOMAIN] shutting down watcher thread
145 [2026-01-11T02:32:07,297][INFO ][o.e.c.f.AbstractFileWatchingService] [MY_DOMAIN] watcher service stopped
146 [2026-01-11T02:32:07,297][INFO ][o.e.x.w.WatcherService   ] [MY_DOMAIN] stopping watch service, reason [shutdown initiated]
147 [2026-01-11T02:32:07,298][INFO ][o.e.x.w.t.s.e.TickerScheduleTriggerEngine] [MY_DOMAIN] Stopping watcher engine
148 [2026-01-11T02:32:07,298][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [MY_DOMAIN] [controller/2566265] [Main.cc@176] ML controller exiting
149 [2026-01-11T02:32:07,298][INFO ][o.e.x.w.WatcherLifeCycleService] [MY_DOMAIN] watcher has stopped and shutdown
150 [2026-01-11T02:32:07,298][INFO ][o.e.x.m.p.NativeController] [MY_DOMAIN] Native controller process has stopped - no new native processes can be started
151 [2026-01-11T02:32:07,790][INFO ][o.e.n.Node               ] [MY_DOMAIN] stopped
152 [2026-01-11T02:32:07,790][INFO ][o.e.n.Node               ] [MY_DOMAIN] closing ...
153 [2026-01-11T02:32:07,801][INFO ][o.e.n.Node               ] [MY_DOMAIN] closed

This means that something sent it a SIGTERM and it shut down gracefully.

This means that systemd is the thing sending it a SIGTERM, because Elasticsearch failed to notify it that it had started up in time. The service file you shared appears to be incomplete or modified from the original so it’s unclear what timeout is configured. But also there’s no loaded module [systemd] in your log which means you’re not using a distribution of Elasticsearch which integrates with systemd (i.e. either a .rpm or .deb package) so the Type=notify thing won’t work. Either use one of the systemd-compatible distributions, or adjust your service file to reflect this.

1 Like

Thank you much!