Unable to start elasticsearch 7.8 container with a password

vorpal_sledge · August 17, 2020, 6:57pm

Hello,
I am attempting to setup an elastisearch cluster using a series of nodes which each will run a docker-compose. However, when starting up the first node I am unable to log in, and see the following error in docker logs:

"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

It also notes that the shards are unavailable and the cluster is going into RED status. My docker-compose file looks like:

services:
       elasticsearch:
               image: -.dkr.ecr.${REGION}.amazonaws.com/-/elasticsearch:latest
               restart: unless-stopped
               ports:
                       - "9200:9200"
                       - "9300:9300"
               environment:
                       - bootstrap.memory_lock=true
                       - node.data=false
                       - cluster.name=${CLUSTER_NAME}
                       - cluster.routing.allocation.awareness.attributes=aws_availability_zone
                       - cluster.initial_master_nodes=${INITIAL_MASTER}
                       - discovery.seed_providers=ec2
                       - discovery.ec2.endpoint=ec2.${REGION}.amazonaws.com
                       - discovery.ec2.tag.Application=ElasticSearchApi
                       - discovery.ec2.tag.DeploymentGroup=${DEPLOYMENTGROUP_TAG}
                       - discovery.ec2.availability_zones=${REGION}a,${REGION}b,${REGION}c
                       - s3.client.default.endpoint=${S3_ENDPOINT}
                       - "ES_JAVA_OPTS=-Xms${MIN_MEM} -Xmx${MAX_MEM}"
                       - xpack.security.enabled=true
                       - xpack.security.transport.ssl.enabled=true
                       - xpack.security.transport.ssl.verification_mode=certificate
                       - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
                       - xpack.security.transport.ssl.certificate=$CERTS_DIR/instance/instance.crt
                       - xpack.security.transport.ssl.key=$CERTS_DIR/instance/instance.key
                       - "ELASTIC_PASSWORD=password"
               volumes:
                       - esdata:/usr/share/elasticsearch/data
                       - /apps/certificates:$CERTS_DIR
               ulimits:
                       nproc: 4096
                       memlock:
                               soft: -1
                               hard: -1
                       nofile:
                               soft: 65536
                               hard: 65536
volumes:
       esdata:

and my .env like

CLUSTER_NAME=elasticsearch-test
DEPLOYMENTGROUP_TAG=test
MIN_MEM=512m
MAX_MEM=512m
REGION=us-east-2
INITIAL_MASTER=10.232.0.166
S3_ENDPOINT=s3.us-east-2.amazonaws.com
CERTS_DIR=/usr/share/elasticsearch/config/certificates
ELASTIC_PASSWORD=password

I am using the 7.8 image from docker hub, pulled down and modified by adding the ec2 discovery and s3 plugins.

warkolm · August 17, 2020, 11:17pm

What does the output from _cat/shards/.security-7 look like?

vorpal_sledge · August 17, 2020, 11:32pm

Unfortunately, I can't even get in to see that as it can't authenticate the elastic user.

{"type": "server", "timestamp": "2020-08-17T23:30:41,959Z", "level": "ERROR", "component": "o.e.x.s.a.e.ReservedRealm", "cluster.name": "elasticsearch-test", "node.name": "9791989c53a4", "message": "failed to retrieve password hash for reserved user [elastic]", "cluster.uuid": "JSSuQtIzTWC1177DJbg6Kg", "node.id": "8RBT-pPgS8akgphCJwCBZw" ,
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable",
"at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:181) ~[x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:525) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:224) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:99) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:167) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:449) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:504) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:416) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:426) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:367) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:325) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:405) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:61) [x-pack-security-7.8.0.jar:7.8.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:329) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:383) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) [elasticsearch-7.8.0.jar:7.8.0]",
"at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:54) [transport-netty4-client-7.8.0.jar:7.8.0]",
"at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:29) [transport-netty4-client-7.8.0.jar:7.8.0]",
"at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.8.0.jar:7.8.0]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]", 
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }

system · September 14, 2020, 11:33pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to start elasticsearch 7.8 container with a password Elasticsearch	2	426	October 11, 2020
New build docker-compose multi-node cluster fails to retrieve password hash for reserved user [elastic] / at least one primary shard for the index [.security-7] is unavailable Elasticsearch elastic-stack-security , docker	4	1906	April 13, 2023
At least one primary shard for the index [.security-7] is unavailable issue Elasticsearch	9	5853	December 5, 2022
Trouble running the ELK-8.3.3 docker-compose instructions. failed to retrieve password hash Elasticsearch docker	1	231	September 15, 2022
Es service not starting up due to error Elasticsearch elastic-stack-security	8	1258	May 24, 2022

Unable to start elasticsearch 7.8 container with a password

Related topics