[2024-06-06T00:16:32,277][ERROR][o.e.b.Elasticsearch ] [server.domain.co.uk] fatal exception while booting Elasticsearch
java.lang.NullPointerException: Cannot invoke "org.elasticsearch.nativeaccess.Systemd.notify_ready()" because "this.systemd" is null
at org.elasticsearch.systemd.SystemdPlugin.notifyReady(SystemdPlugin.java:99) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:113) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194) ~[?:?]
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611) ~[?:?]
at org.elasticsearch.node.Node.start(Node.java:435) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:462) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:255) ~[elasticsearch-8.14.0.jar:?]
I encountered the same error when I upgraded Elasticsearch from 8.13.4 to 8.14.0.
Upon checking /var/log/elasticsearch/elasticsearch.log, I found message Caused by: java.nio.file.AccessDeniedException: /usr/lib64/mysql/plugin/auth_pam_tool_dir.
I resolved it by using the command setfacl -m u:elasticsearch:rx /usr/lib64/mysql/plugin/auth_pam_tool_dir.
1 Like
Had the same myself. I had to run the same command for mysql, and also for redis where that was installed on the server also.
Same here, in my case there were additional directories I was getting AccessDeniedException errors, so I had to do:
setfacl -m u:elasticsearch:rx /usr/lib64/mysql/plugin/auth_pam_tool_dir
setfacl -m u:elasticsearch:rx /usr/lib64/redis
setfacl -m u:elasticsearch:rx /usr/lib64/redis/modules
setfacl -m u:elasticsearch:rx /usr/lib/Acronis/AgentCommData
Ah, apologies you're encountering problems upgrading Elasticsearch. Could you post the full stack trace for the AccessDeniedException in the log so we can look closer at what's going on here?
Caused by: java.nio.file.AccessDeniedException: /usr/lib64/redis/modules
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.asIOException(UnixException.java:115) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:477) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:481) ~[?:?]
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301) ~[?:?]
at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374) ~[?:?]
at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83) ~[?:?]
at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611) ~[?:?]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:622) ~[?:?]
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:631) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:637) ~[?:?]
at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:642) ~[?:?]
at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.findLibSystemd(JdkSystemdLibrary.java:74) ~[elasticsearch-native-8.14.0.jar:?]
at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.<clinit>(JdkSystemdLibrary.java:36) ~[elasticsearch-native-8.14.0.jar:?]
... 7 more
Caused by: java.nio.file.AccessDeniedException: /usr/lib64/mysql/plugin/auth_pam_tool_dir
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.asIOException(UnixException.java:115) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:477) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:481) ~[?:?]
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301) ~[?:?]
at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374) ~[?:?]
at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83) ~[?:?]
at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611) ~[?:?]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:622) ~[?:?]
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:631) ~[?:?]
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:637) ~[?:?]
at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:642) ~[?:?]
at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.findLibSystemd(JdkSystemdLibrary.java:74) ~[elasticsearch-native-8.14.0.jar:?]
at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.<clinit>(JdkSystemdLibrary.java:36) ~[elasticsearch-native-8.14.0.jar:?]
... 7 more
The team is working on a fix for it for 8.14.1. I don't have any release date yet.
Note that this problem does not appear when running on Docker or on cloud.elastic.co.
[2024-06-06T13:56:47,184][INFO ][o.e.t.a.APM ] [ns537244.ip-139-99-131.net] Sending apm metrics is disabled
[2024-06-06T13:56:47,184][INFO ][o.e.t.a.APM ] [ns537244.ip-139-99-131.net] Sending apm tracing is disabled
[2024-06-06T13:56:47,203][INFO ][o.e.x.s.Security ] [ns537244.ip-139-99-131.net] Security is enabled
[2024-06-06T13:56:47,455][INFO ][o.e.x.s.a.s.FileRolesStore] [ns537244.ip-139-99-131.net] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-06-06T13:56:47,767][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [ns537244.ip-139-99-131.net] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2024-06-06T13:56:47,946][INFO ][o.e.x.w.Watcher ] [ns537244.ip-139-99-131.net] Watcher initialized components at 2024-06-06T13:56:47.946Z
[2024-06-06T13:56:47,991][INFO ][o.e.x.p.ProfilingPlugin ] [ns537244.ip-139-99-131.net] Profiling is enabled
[2024-06-06T13:56:48,004][INFO ][o.e.x.p.ProfilingPlugin ] [ns537244.ip-139-99-131.net] profiling index templates will not be installed or reinstalled
[2024-06-06T13:56:48,007][INFO ][o.e.x.a.APMPlugin ] [ns537244.ip-139-99-131.net] APM ingest plugin is disabled
[2024-06-06T13:56:48,356][INFO ][o.e.t.n.NettyAllocator ] [ns537244.ip-139-99-131.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2024-06-06T13:56:48,376][INFO ][o.e.i.r.RecoverySettings ] [ns537244.ip-139-99-131.net] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-06-06T13:56:48,438][INFO ][o.e.d.DiscoveryModule ] [ns537244.ip-139-99-131.net] using discovery type [multi-node] and seed hosts providers [settings]
[2024-06-06T13:56:49,399][INFO ][o.e.n.Node ] [ns537244.ip-139-99-131.net] initialized
[2024-06-06T13:56:49,399][INFO ][o.e.n.Node ] [ns537244.ip-139-99-131.net] starting ...
[2024-06-06T13:56:49,466][INFO ][o.e.x.s.c.f.PersistentCache] [ns537244.ip-139-99-131.net] persistent cache index loaded
[2024-06-06T13:56:49,467][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [ns537244.ip-139-99-131.net] deprecation component started
[2024-06-06T13:56:49,524][INFO ][o.e.t.TransportService ] [ns537244.ip-139-99-131.net] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2024-06-06T13:56:50,426][WARN ][o.e.c.c.ClusterBootstrapService] [ns537244.ip-139-99-131.net] this node is locked into cluster UUID [Mtu4GqNfSGKN-AFIB8ovQA] but [cluster.initial_master_nodes] is set to [ns537244.ip-139-99-131.net]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/important-settings.html#initial_master_nodes
[2024-06-06T13:56:50,652][INFO ][o.e.c.s.MasterService ] [ns537244.ip-139-99-131.net] elected-as-master ([1] nodes joined in term 44)[_FINISH_ELECTION_, {ns537244.ip-139-99-131.net}{7fEFDqDjTO6_atywhRG79Q}{dAiyBk1XQParkFqheg7-mw}{ns537244.ip-139-99-131.net}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.14.0}{7000099-8505000} completing election], term: 44, version: 2964, delta: master node changed {previous [], current [{ns537244.ip-139-99-131.net}{7fEFDqDjTO6_atywhRG79Q}{dAiyBk1XQParkFqheg7-mw}{ns537244.ip-139-99-131.net}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.14.0}{7000099-8505000}]}
[2024-06-06T13:56:50,864][INFO ][o.e.c.s.ClusterApplierService] [ns537244.ip-139-99-131.net] master node changed {previous [], current [{ns537244.ip-139-99-131.net}{7fEFDqDjTO6_atywhRG79Q}{dAiyBk1XQParkFqheg7-mw}{ns537244.ip-139-99-131.net}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.14.0}{7000099-8505000}]}, term: 44, version: 2964, reason: Publication{term=44, version=2964}
[2024-06-06T13:56:50,884][INFO ][o.e.c.f.AbstractFileWatchingService] [ns537244.ip-139-99-131.net] starting file watcher ...
[2024-06-06T13:56:50,887][INFO ][o.e.c.f.AbstractFileWatchingService] [ns537244.ip-139-99-131.net] file settings service up and running [tid=78]
[2024-06-06T13:56:50,887][INFO ][o.e.r.s.FileSettingsService] [ns537244.ip-139-99-131.net] setting file [/etc/elasticsearch/operator/settings.json] not found, initializing [file_settings] as empty
[2024-06-06T13:56:50,891][INFO ][o.e.c.c.NodeJoinExecutor ] [ns537244.ip-139-99-131.net] node-join: [{ns537244.ip-139-99-131.net}{7fEFDqDjTO6_atywhRG79Q}{dAiyBk1XQParkFqheg7-mw}{ns537244.ip-139-99-131.net}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.14.0}{7000099-8505000}] with reason [completing election]
[2024-06-06T13:56:50,891][INFO ][o.e.h.AbstractHttpServerTransport] [ns537244.ip-139-99-131.net] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2024-06-06T13:56:50,900][INFO ][o.e.n.Node ] [ns537244.ip-139-99-131.net] started {ns537244.ip-139-99-131.net}{7fEFDqDjTO6_atywhRG79Q}{dAiyBk1XQParkFqheg7-mw}{ns537244.ip-139-99-131.net}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.14.0}{7000099-8505000}{ml.allocated_processors_double=8.0, ml.max_jvm_size=5368709120, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=33465880576, ml.allocated_processors=8}
[2024-06-06T13:56:50,901][ERROR][o.e.b.Elasticsearch ] [ns537244.ip-139-99-131.net] fatal exception while booting Elasticsearch
java.lang.NullPointerException: Cannot invoke "org.elasticsearch.nativeaccess.Systemd.notify_ready()" because "this.systemd" is null
at org.elasticsearch.systemd.SystemdPlugin.notifyReady(SystemdPlugin.java:99) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:113) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194) ~[?:?]
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611) ~[?:?]
at org.elasticsearch.node.Node.start(Node.java:435) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:462) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:255) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.14.0.jar:?]
[2024-06-06T13:56:50,906][INFO ][o.e.n.Node ] [ns537244.ip-139-99-131.net] stopping ...
[2024-06-06T13:56:50,906][INFO ][o.e.c.f.AbstractFileWatchingService] [ns537244.ip-139-99-131.net] shutting down watcher thread
[2024-06-06T13:56:50,907][INFO ][o.e.c.f.AbstractFileWatchingService] [ns537244.ip-139-99-131.net] watcher service stopped
[2024-06-06T13:56:50,909][INFO ][o.e.x.w.WatcherService ] [ns537244.ip-139-99-131.net] stopping watch service, reason [shutdown initiated]
[2024-06-06T13:56:50,909][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ns537244.ip-139-99-131.net] [controller/1963869] [Main.cc@176] ML controller exiting
[2024-06-06T13:56:50,909][INFO ][o.e.x.m.p.NativeController] [ns537244.ip-139-99-131.net] Native controller process has stopped - no new native processes can be started
[2024-06-06T13:56:50,909][INFO ][o.e.x.w.WatcherLifeCycleService] [ns537244.ip-139-99-131.net] watcher has stopped and shutdown
[2024-06-06T13:56:50,943][WARN ][o.e.s.SnapshotsService ] [ns537244.ip-139-99-131.net] failed to update snapshot state after shards started or nodes removed from [update snapshot after shards changed [false] or node configuration changed [true]]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node closed
at org.elasticsearch.cluster.service.MasterService.forkQueueProcessor(MasterService.java:1292) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.onCompletion(MasterService.java:1256) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$5$1.onResponse(MasterService.java:1219) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$5$1.onResponse(MasterService.java:1216) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:269) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$2.onResponse(MasterService.java:284) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$2.onResponse(MasterService.java:281) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$3.run(MasterService.java:451) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:280) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.AbstractThreadedActionListener$1.onRejection(AbstractThreadedActionListener.java:52) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onRejection(ThreadContext.java:972) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:82) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.AbstractThreadedActionListener.onFailure(AbstractThreadedActionListener.java:37) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:2064) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:394) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:306) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:205) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1961) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:119) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$ApplyCommitResponseHandler.onFailure(Publication.java:427) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Coordinator$7.onFailure(Coordinator.java:1670) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportResponseHandler$1.handleException(TransportResponseHandler.java:63) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1476) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1476) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:986) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:244) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:979) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:945) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$2.sendRequest(SecurityServerTransportInterceptor.java:273) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:457) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.lambda$sendRequestInner$0(SecurityServerTransportInterceptor.java:210) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:170) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:179) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendRequestInner(SecurityServerTransportInterceptor.java:208) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:178) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:849) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:759) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.sendApplyCommit(Coordinator.java:2168) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$1.onResponse(Publication.java:291) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$1.onResponse(Publication.java:288) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Coordinator$7.onResponse(Coordinator.java:1663) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:382) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:302) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:205) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:170) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget.sendApplyCommit(Publication.java:288) ~[elasticsearch-8.14.0.jar:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611) ~[?:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget.lambda$handlePublishResponse$0(Publication.java:275) ~[elasticsearch-8.14.0.jar:?]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget.handlePublishResponse(Publication.java:269) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onResponse(Publication.java:380) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onResponse(Publication.java:357) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.coordination.Coordinator$7.onResponse(Coordinator.java:1663) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:57) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:54) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: master service is in state [STOPPED]
at org.elasticsearch.cluster.service.MasterService.getRejectionException(MasterService.java:1306) ~[elasticsearch-8.14.0.jar:?]
... 76 more
[2024-06-06T13:56:50,954][INFO ][o.e.n.Node ] [ns537244.ip-139-99-131.net] stopped
[2024-06-06T13:56:50,954][INFO ][o.e.n.Node ] [ns537244.ip-139-99-131.net] closing ...
In my case it was a cPanel server (v120.0.9) running AlmaLinux (v8.10).
Are you guys also running similar environments?
We have created a knowledge base article for customers and updated the release notes with some workarounds you can use until we release 8.14.1 to fix this issue.
Thanks for reporting it, and it'll be fixed in 8.14.1.
I can't access this knowledge base article, can you please tell me why ? its asking for login, maybe its private something.
Yeah, I'm afraid that's internal / customers only. But the relevant info is: Add systemd native access by rjernst · Pull Request #106151 · elastic/elasticsearch · GitHub is getting backported to 8.14.1 and will then get a fast release. So hold tight.
In the meantime, Elasticsearch version 8.14.0 | Elasticsearch Guide [8.14] | Elastic has a workaround.
And sorry for the trouble!
2 Likes
In my case I did not get any permission errors (at least in the log) so I don't know where to change any permissions. But I found out, that the problem only occurs if I try to run elastic in daemon mode. Starting elastic on console works without any problems. So my workaround is to start the process within a screen session as long as the bugfix release is available.
root@abc:~# cd /usr/share/elasticsearch/
root@abc:/usr/share/elasticsearch# screen -S elastic
root@abc:/usr/share/elasticsearch# sudo -u elasticsearch bin/elasticsearch
After that, detach session with CTRL+a d.
1 Like
@xeraa shouldn't the 8.14.0 rpm/deb/tar.gz packages be removed from download then?
Unfortunately a lot of people do not even look at release notes before upgrading and this breaks the upgrade.
1 Like
This worked for me as well. Thanks for sharing
It looks like findLibSystemd() walks through all directories passed by java.library.path when starting elasticsearch, e.g. /usr/lib which has a lot of files (Debian 12)...
I had a lot of __pycache__ folders in there which I just deleted with find /usr/lib | grep -E "/__pycache__$" | xargs rm -rf.
To find other files/directories without world read permissions:
- List them:
find /usr/lib ! -perm -o+r - Set the permissions:
find /usr/lib -type d ! -perm -o+r -exec chmod o+rx {} \; && find /usr/lib -type f ! -perm -o+r -exec chmod o+r {} \;
Since there is a straight forward workaround, I don't think there are any immediate plans to pull the download. Though the "current" one will soon change.
1 Like
8.14.1 is out with the fix now: Elasticsearch version 8.14.1 | Elasticsearch Guide [8.14] | Elastic
1 Like