Elasticsearch 8.11.1 fails when using AWS IAM roles for service accounts

mornie · November 22, 2023, 3:58pm

Hi Team

I am using eck-operator-2.10.0, and when upgrading the elastic stack from 8.11.0 to 8.11.1, the elasticsearch pods is stuck in an crashloopbackoff state.

I have configured AWS IAM roles for service accounts (IRSA) to create automated snapshots. It's configured in the same way as described on this page:

When removing this configuration it starts up without issues, but as soon as it's added back, it fails again with the following error from the Elasticsearch container:

{"@timestamp":"2023-11-22T15:49:33.417Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elastic-stack-dev-es-data-nodes-0","elasticsearch.cluster.name":"elastic-stack-dev","error.type":"java.security.AccessControlException","error.message":"access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")","error.stack_trace":"java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1071)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)\n\tat java.base/java.lang.Class.checkMemberAccess(Class.java:3227)\n\tat java.base/java.lang.Class.getDeclaredConstructors(Class.java:2725)\n\tat com.fasterxml.jackson.databind.util.ClassUtil.getConstructors(ClassUtil.java:1331)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialConstructors(AnnotatedCreatorCollector.java:115)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:70)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collectCreators(AnnotatedCreatorCollector.java:61)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedClass._creators(AnnotatedClass.java:403)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedClass.getFactoryMethods(AnnotatedClass.java:315)\n\tat com.fasterxml.jackson.databind.introspect.BasicBeanDescription.getFactoryMethods(BasicBeanDescription.java:573)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitFactoryCreators(BasicDeserializerFactory.java:641)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:278)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:222)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:1421)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:403)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:644)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:644)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:621)\n\tat com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:188)\n\tat com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:28)\n\tat com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:836)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:550)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:654)\n\tat com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4956)\n\tat com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4826)\n\tat com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3809)\n\tat com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:92)\n\tat com.amazonaws.partitions.PartitionsLoader.build(PartitionsLoader.java:84)\n\tat com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30)\n\tat com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64)\n\tat com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52)\n\tat com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:106)\n\tat com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256)\n\tat com.amazonaws.client.builder.AwsClientBuilder.withRegion(AwsClientBuilder.java:245)\n\tat org.elasticsearch.repositories.s3.S3Service$CustomWebIdentityTokenCredentialsProvider.<init>(S3Service.java:373)\n\tat org.elasticsearch.repositories.s3.S3Service.<init>(S3Service.java:98)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.s3Service(S3RepositoryPlugin.java:115)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.createComponents(S3RepositoryPlugin.java:109)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.lambda$new$17(Node.java:759)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:263)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.<init>(Node.java:775)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.<init>(Node.java:344)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:236)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:236)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73)\n"}
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elastic-stack-dev.log

The issue seems to be similar with this issue being reported: Elasticsearch crashes on startup when upgrading from 8.10.4 to 8.11.1 when S3 snapshots are in use · Issue #102173 · elastic/elasticsearch · GitHub

As suggested in this issue, I have tried to set AWS_STS_REGIONAL_ENDPOINTS to an empty string in the Stateful Set, which does redeploy the Elasticsearch containers and sets the container with AWS_STS_REGIONAL_ENDPOINTS as an empty string as an environment variable.

However it still fails to start-up and keeps getting crashloopbackoff.

Environment

ECK version:
2.10.0
Kubernetes information
- Kubernetes distribution: EKS on AWS (1.28)

Any ideas?

toonvanstrijp · November 28, 2023, 11:01am

I'm facing the exact same issue. I use the following script to create a symbolic link from the AWS_WEB_IDENTITY_TOKEN_FILE to the config dir in elasticsearch.

#!/bin/bash
# Load environment
. /opt/bitnami/scripts/elasticsearch-env.sh

mkdir -p "$ELASTICSEARCH_CONF_DIR/repository-s3"
    
ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "$ELASTICSEARCH_CONF_DIR/repository-s3/aws-web-identity-token-file"

system · December 26, 2023, 11:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.