The stack cannot be started according to the instructions

Elastic Stack Elasticsearch
1

I'm trying to run the stack in docker according to this instruction.

My .env:

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=elastic

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=kibana

# Version of Elastic products
STACK_VERSION=8.6.2

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject

At the kibana download stage, I get the following error:

[2023-03-10T09:24:11.109+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

And there are such errors in the elasticsearch container:

{"@timestamp":"2023-03-10T09:25:53.605Z", "log.level":"ERROR", "message":"failed to retrieve password hash for reserved user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.security.authc.esnative.ReservedRealm","trace.id":"6182cca2752e51bd81d2428a7a29c162","elasticsearch.cluster.uuid":"kiUrFIPdTKCDlKzo75K_VA","elasticsearch.node.id":"tK3ZqWc6T5qp18M-lqIEhw","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"at least one primary shard for the index [.security-7] is unavailable","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:138)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:603)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:275)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:135)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:147)\n\tat org.elasticsearch.xcore@8.6.2/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:234)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:83)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:180)\n\tat org.elasticsearch.xcore@8.6.2/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158)\n\tat org.elasticsearch.xcore@8.6.2/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158)\n\tat org.elasticsearch.xcore@8.6.2/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:158)\n\tat org.elasticsearch.xcore@8.6.2/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:136)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:95)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127)\n\tat org.elasticsearch.security@8.6.2/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:101)\n\tat org.elasticsearch.server@8.6.2/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:398)\n\tat org.elasticsearch.server@8.6.2/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:532)\n\tat org.elasticsearch.server@8.6.2/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:313)\n\tat org.elasticsearch.server@8.6.2/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:379)\n\tat org.elasticsearch.server@8.6.2/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:460)\n\tat org.elasticsearch.server@8.6.2/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:353)\n\tat org.elasticsearch.transport.netty4@8.6.2/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128)\n\tat org.elasticsearch.transport.netty4@8.6.2/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler@4.1.84.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)\n\tat io.netty.handler@4.1.84.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)\n\tat io.netty.handler@4.1.84.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458)\n\tat io.netty.codec@4.1.84.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.84.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.84.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.84.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n"}
{"@timestamp":"2023-03-10T09:25:53.608Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"6182cca2752e51bd81d2428a7a29c162","elasticsearch.cluster.uuid":"kiUrFIPdTKCDlKzo75K_VA","elasticsearch.node.id":"tK3ZqWc6T5qp18M-lqIEhw","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}

If I do everything according to the instructions and it does not work, then what did I not take into account? Or is the instruction incorrect?

2

Welcome!

What exact command line are you running? What is your docker-compose.yml file?

3

I copied the example given here into my docker-compose.yml

изображение

Everything went well, but everything slows down on kibana authorization

4

Did you modify anything in that file?

5

I just tested it locally and it's all good.

6

I recorded a video of starting the stack from the very beginning. I have Alpine Linux running on which I run. Docker works on it.

7

Maybe I should look at some logs to see the reason?

8

Hi @alexanderzhirov

There's also a kind of a subtle issue.

If you're just testing restarting etc, you should cleanup / delete the volumes because the volumes do not get deleted each time you run this and if they already exist.... they don't get written over. So if for some reason it didn't work the first time running it over and over typically won't fix it.

I've had several users running to this and it was very tricky to find.

9

I delete volumes before starting. In the video I demonstrated the launch on a clean docker. I have a feeling that docker does not read the set password from the .env file

10

Did you try with a simple password, perhaps you have a character that's having trouble being read (ohh I see simple).. ok I'll take a look at the video... The process works for me as well a couple days ago I will test again.

Oh @alexanderzhirov Welcome to the community....

1 Like
11

So I reran from scratch and when I say scratch I mean reallly scratch

I ran a docker system prune -a

It ran fine Here is my output.

$ docker-compose up -d
..... Lots of images being pulled
   ⠿ b2180a238095 Pull complete                                                                                     98.0s
[+] Running 6/6
 ⠿ Network test_default     Created                                                                                  0.1s
 ⠿ Container test-setup-1   Healthy                                                                                  2.4s
 ⠿ Container test-es01-1    Healthy                                                                                 74.1s
 ⠿ Container test-es02-1    Healthy                                                                                 65.1s
 ⠿ Container test-es03-1    Healthy                                                                                 66.1s
 ⠿ Container test-kibana-1  Started                                                                                 74.3s

Everything came up and work, I even logged

I watched the video.

I did notice that you did not clean up the volumes before trying the second time.

Also you should try to clean the container images... I did not see them get pulled, perhaps you have some leftover containers...

Also noticed thatnetwork and volume drivers look different, the output looks different etc .... I am not sure if portainer.io is something different

Also after you clean all up and try again ... making sure the volumes on both the docker and host side are cleaned up try this

$ curl -k -u elastic:elastic https://localhost:9200
{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "A9Uu3dPjTJWiJZ8hRkL1gA",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
1 Like
12

Damn, you were right! It was necessary to perform a complete cleaning. I have accumulated 5 gigabytes in my cache. After deleting the cache, everything worked out! Thank you very much! My inattention cost me a lesson. :grinning:

1 Like
13

Yeah I have used docker a lot but I am not an expert... sometimes it is unclear to me what is left behind and what is not... lucky for me I can run a prune and start clean.

Cool enjoy...

Now what is interesting ... you should really may a copy of the compose without all the setup stuff ... because no need to run all that over and over... if that make sense.... it is really just meant as a startup example...

14

I'm just trying to get acquainted with this stack through training examples :man_student:

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.