# Elasticsearch 8.11.1 fails when using AWS IAM roles for service accounts **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [November 22, 2023, 3:58pm UTC](https://discuss.elastic.co/t/elasticsearch-8-11-1-fails-when-using-aws-iam-roles-for-service-accounts/347766 "2023-11-22T15:58:46Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![mornie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mornie/32/127929_2.png) [@mornie](https://discuss.elastic.co/u/mornie)\ **Post date:** [November 22, 2023, 3:58pm UTC](https://discuss.elastic.co/t/elasticsearch-8-11-1-fails-when-using-aws-iam-roles-for-service-accounts/347766/1 "2023-11-22T15:58:46Z")
Hi Team I am using eck-operator-2.10.0, and when upgrading the elastic stack from 8.11.0 to 8.11.1, the elasticsearch pods is stuck in an crashloopbackoff state. I have configured AWS IAM roles for service accounts (IRSA) to create automated snapshots. It's configured in the same way as described on this page: > **[Create automated snapshots | Elastic Cloud on Kubernetes \[2.10\] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-snapshots.html#k8s-iam-service-accounts)** When removing this configuration it starts up without issues, but as soon as it's added back, it fails again with the following error from the Elasticsearch container: ```auto {"@timestamp":"2023-11-22T15:49:33.417Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elastic-stack-dev-es-data-nodes-0","elasticsearch.cluster.name":"elastic-stack-dev","error.type":"java.security.AccessControlException","error.message":"access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")","error.stack_trace":"java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1071)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)\n\tat java.base/java.lang.Class.checkMemberAccess(Class.java:3227)\n\tat java.base/java.lang.Class.getDeclaredConstructors(Class.java:2725)\n\tat com.fasterxml.jackson.databind.util.ClassUtil.getConstructors(ClassUtil.java:1331)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialConstructors(AnnotatedCreatorCollector.java:115)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:70)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collectCreators(AnnotatedCreatorCollector.java:61)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedClass._creators(AnnotatedClass.java:403)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedClass.getFactoryMethods(AnnotatedClass.java:315)\n\tat com.fasterxml.jackson.databind.introspect.BasicBeanDescription.getFactoryMethods(BasicBeanDescription.java:573)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitFactoryCreators(BasicDeserializerFactory.java:641)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:278)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:222)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:1421)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:403)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:644)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:644)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:621)\n\tat com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:188)\n\tat com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:28)\n\tat com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:836)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:550)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:654)\n\tat com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4956)\n\tat com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4826)\n\tat com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3809)\n\tat com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:92)\n\tat com.amazonaws.partitions.PartitionsLoader.build(PartitionsLoader.java:84)\n\tat com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30)\n\tat com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64)\n\tat com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52)\n\tat com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:106)\n\tat com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256)\n\tat com.amazonaws.client.builder.AwsClientBuilder.withRegion(AwsClientBuilder.java:245)\n\tat org.elasticsearch.repositories.s3.S3Service$CustomWebIdentityTokenCredentialsProvider.(S3Service.java:373)\n\tat org.elasticsearch.repositories.s3.S3Service.(S3Service.java:98)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.s3Service(S3RepositoryPlugin.java:115)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.createComponents(S3RepositoryPlugin.java:109)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.lambda$new$17(Node.java:759)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:263)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.(Node.java:775)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.(Node.java:344)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:236)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:236)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73)\n"} ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elastic-stack-dev.log ``` The issue seems to be similar with this issue being reported: [Elasticsearch crashes on startup when upgrading from 8.10.4 to 8.11.1 when S3 snapshots are in use · Issue #102173 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/102173) As suggested in this issue, I have tried to set **AWS\_STS\_REGIONAL\_ENDPOINTS** to an empty string in the Stateful Set, which does redeploy the Elasticsearch containers and sets the container with AWS\_STS\_REGIONAL\_ENDPOINTS as an empty string as an environment variable. However it still fails to start-up and keeps getting crashloopbackoff. **Environment** - ECK version: 2.10.0 - Kubernetes information - Kubernetes distribution: EKS on AWS (1.28) Any ideas? ---
**Author:** ![toonvanstrijp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toonvanstrijp/32/128129_2.png) [@toonvanstrijp](https://discuss.elastic.co/u/toonvanstrijp)\ **Post date:** [November 28, 2023, 11:01am UTC](https://discuss.elastic.co/t/elasticsearch-8-11-1-fails-when-using-aws-iam-roles-for-service-accounts/347766/2 "2023-11-28T11:01:59Z")
I'm facing the exact same issue. I use the following script to create a symbolic link from the AWS\_WEB\_IDENTITY\_TOKEN\_FILE to the config dir in elasticsearch. ```auto #!/bin/bash # Load environment . /opt/bitnami/scripts/elasticsearch-env.sh mkdir -p "$ELASTICSEARCH_CONF_DIR/repository-s3" ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "$ELASTICSEARCH_CONF_DIR/repository-s3/aws-web-identity-token-file" ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 26, 2023, 11:02am UTC](https://discuss.elastic.co/t/elasticsearch-8-11-1-fails-when-using-aws-iam-roles-for-service-accounts/347766/3 "2023-12-26T11:02:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.