Service killing each week - Error loading SPI class list from URL - x-pack-security

Elastic Stack Elasticsearch
1

Hi :smiley:

I was using Elastic Search 8.14.2, and the service would crash every week. The log output showed that x-pack-security-8.14.1 was not found, but the installed Elastic Search version was 8.14.2. I tried reinstalling Elastic Search, removing all old files and configurations. I installed version 8.14.3, but the same problem occurred, and now it complains about the x-pack-security version 8.14.2. Has anyone experienced this? Any tips?
The log output:

[2024-07-14T01:30:00,000][INFO ][o.e.x.s.SnapshotRetentionTask] [elastic-vm] starting SLM retention snapshot cleanup task

[2024-07-14T01:30:00,000][INFO ][o.e.x.m.MlDailyMaintenanceService] [elastic-vm] triggering scheduled [ML] maintenance tasks

[2024-07-14T01:30:00,005][INFO ][o.e.x.s.SnapshotRetentionTask] [elastic-vm] there are no repositories to fetch, SLM retention snapshot cleanup task complete

[2024-07-14T01:30:00,006][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [elastic-vm] Deleting expired data

[2024-07-14T01:30:00,009][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [elastic-vm] Successfully deleted [0] unused stats documents

[2024-07-14T01:30:00,009][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [elastic-vm] Completed deletion of expired ML data

[2024-07-14T01:30:00,010][INFO ][o.e.x.m.MlDailyMaintenanceService] [elastic-vm] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask

[2024-07-14T02:52:37,016][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elastic-vm] fatal error in thread [elasticsearch[elastic-vm][masterService#updateTask][T#970]], exiting

java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.14.2.jar!/META-INF/services/org.elas

ticsearch.index.SlowLogFieldProvider

at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1399) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:742) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:699) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:471) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestForSystemIndex(MetadataCreateIndexService.java:711) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:374) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:439) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$CreateIndexTask.execute(AutoCreateAction.java:334) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction.lambda$new$0(AutoCreateAction.java:120) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1645) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1642) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.2.jar:?]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.2.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]

at java.lang.Thread.run(Thread.java:1570) ~[?:?]

Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.14.2.jar

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]

at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?]

at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]

at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]

at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?]

at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?]

at java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:724) ~[?:?]

at java.util.zip.ZipFile.<init>(ZipFile.java:251) ~[?:?]

at java.util.zip.ZipFile.<init>(ZipFile.java:180) ~[?:?]

at java.util.jar.JarFile.<init>(JarFile.java:345) ~[?:?]

at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:100) ~[?:?]

at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?]

at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168) ~[?:?]

at sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91) ~[?:?]

at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:117) ~[?:?]

at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:160) ~[?:?]

at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:112) ~[elasticsearch-8.14.2.jar:?]

... 26 more
2

hi alan

I also have the same issue in version 8.17.1 . can you guide me for a solution.
thanks

3

This happens when elasticsearch gets an upgrade and elasticsearch is still running. The old x-pack-security-8.14.2.jar file gets deleted and a file with name x-pack-security-8.14.3.jar was created during the upgrade process.

The still running elasticsearch process tries to access the old file and crashes.
Do you have an automatic job for updating your packages like unattended-upgrades for Ubuntu? This could be the reason.

To me it looks like it only happens every know and then. Maybe if someone authenticates against Elasticsearch after upgrading the version?

A solution could be to exclude automatic elasticsearch upgrades, or change your systemd service file to restart elasticsearch after a crash by putting Restart=on-failure to your service file.