Authentication of [elastic] was terminated by realm [reserved]

Akanksha_Pandey · June 6, 2021, 3:34pm

My ELK stack version is 7.9.2

My aim is to enable authentication in Kibana login page but I'm not able to do that. I'm alays directed to the dashboard directly and it never asks for username and password

ELK servers are running on Docker containers.
In my docker-compose.yaml file elasticsearch.yml file mapping is from conf directory i.e ./elasticsearch/conf/elasticsearch.yml

and I had enabled x-pack.security: true in kibana.yml, logstash.yml and elasticserach.yml as well as docker-compose.yaml

When I tried to setup passwords by going inside the elasticserach containers and navigated to bin directory and tried running script elasticsearch-setup-passwords interactive it gave me the below error


Unexpected response code [500] from calling GET http://172.21.0.2:9200/_security/_authenticate?pretty
It doesn't look like the X-Pack security feature is enabled on this Elasticsearch node.
Please check if you have enabled X-Pack security in your elasticsearch.yml configuration file.

ERROR: X-Pack Security is disabled by configuration.

As a result I tried enabling x-pack security inside elasticsearch config folder using the below command-

xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

and restarted the conatiners again- it started giving me below errors-

[INFO ][o.e.x.s.a.AuthenticationService] [inlxdev02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

and this is my docker-compose.yaml file:

version: '3.7'
services: 
  elasticsearch: 
    build:
      context: elasticsearch/
    container_name: elasticsearch
    volumes:
      - type: bind
        source: ./elasticsearch/conf/elasticsearch.yml
        target: /usr/share/elasticsearch/conf/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    environment:
      ES_JAVA_OPTS: "-Xmx2g -Xms2g"
      ELASTIC_PASSWORD: password
      ELASTIC_USERNAME : username
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
    networks:
      - elastic
    restart: always
  
  logstash:
    container_name: logstash
    build: 
      context: logstash/
    # command: logstash -f /conf/logstash.conf
    volumes:
      - type: bind
        source: ./logstash/conf/logstash.yml
        target: /usr/share/logstash/conf/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
     - "5043:5043"
     - "5044:5044"
    environment:
      LS_JAVA_OPTS: "-Xmx1g -Xms1g"
    networks:
      - elastic
    depends_on:
     - elasticsearch
    restart: always

  kibana:
    build:
      context: kibana/
    container_name: vimaan-kibana
    volumes:
      - type: bind
        source: ./kibana/conf/kibana.yml
        target: /usr/share/kibana/conf/kibana.yml
        read_only: true
    ports:
      - "5601:5601"
    networks:
        - elastic
    depends_on:
      - elasticsearch
    restart: always

volumes:
  elasticsearch:

networks:
  elastic:
    driver: bridge

Please help me @TimV @stephenb

Akanksha_Pandey · June 6, 2021, 4:08pm

Also- this is my Kibana UI response

Why is my elasticsearch.yml file settings not reflecting? @stephenb @TimV

warkolm · June 7, 2021, 1:35am

Please don't post pictures of text or code or logs. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

stephenb · June 7, 2021, 2:37am

Perhaps you should try setting this up without docker first just download the tar.gz or zip of elasticsearch and kibana and get it working the way you want. Then replicate with docker.

Also The path in docker to the elasticsearch.yml is not correct, so it is probably not being found. I think perhaps you need to be a little more careful with path, rebuilding containers etc..

Correct
/usr/share/elasticsearch/config/elasticsearch.yml
not
/usr/share/elasticsearch/conf/elasticsearch.yml

You should probably double check the other paths as well.

Kibana path is not correct either.

Logstash I don't have running...

You can use the following command to look into the containers.

docker elasticsearch -it es01 /bin/bash

Akanksha_Pandey · June 7, 2021, 8:38am

@stephenb thanks a lot for your response! I was also able to figure out the root cause- path configuration.

Now, I'm able to get the login page on Kibana and everything works fine except one thing- in my elasticsearch server I'm still getting the authentication error and I'm not getting the reason-

{"type": "server", "timestamp": "2021-06-07T01:16:21,457Z", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "docker-cluster", "node.name": "fcf49d1a87d0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "bSnpaUuRQ0yoAhWoERU31A", "node.id": "Zj-mswZjSX-CnRKUUFkYKg"  }
{"type": "server", "timestamp": "2021-06-07T01:16:22,346Z", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "docker-cluster", "node.name": "fcf49d1a87d0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "bSnpaUuRQ0yoAhWoERU31A", "node.id": "Zj-mswZjSX-CnRKUUFkYKg"  }

And below is the logstash logs-

[2021-06-07T08:36:07,429][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
[2021-06-07T08:36:07,429][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}

Akanksha_Pandey · June 7, 2021, 9:22am

@warkolm I have provided the codes in proper format. The reason for posting snapshots/images were for references only and they are optional. Still I'll keep that in mind from next time. Thanks for your suggestion

stephenb · June 7, 2021, 1:00pm

401 Authentication

username and or password is incorrect

try from the command line

curl -u username:password http://localhost:9200

You will probably get the same error, if so you have not setup the passwords correct.

Akanksha_Pandey · June 7, 2021, 1:29pm

@stephenb Before deploying I changed the password for elastic superuser and got success message as well using elasticsearch-setup-passwords interactive

now, also I again changed it in Kibana devtool by using this command and got success as well but still getting the same authentication errors-

POST /_security/user/elastic/_password
{
"password" : "my_password"
}

Akanksha_Pandey · June 7, 2021, 1:31pm

@stephenb and if the username and password was wrong how I'm able to login in Kibana UI and access it?

stephenb · June 7, 2021, 1:37pm

I don't know.. but 401 is an authentication request...

You do not run the commands / steps I request so I can not help. I ask questions for a specific reason.

Akanksha_Pandey · June 7, 2021, 1:41pm

sorry about that. Can you help me where to run the curl command or it's equivalent request in kibana devtools?

Akanksha_Pandey · June 7, 2021, 1:46pm

@stephenb I ran the curl command inside elasticsearch container and got this response-

"name" : "15ff94f4yediwa",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "bcwouhwWOvsuoERU31A",
  "version" : {
    "number" : "7.9.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "mciorqu4jdk0wm3e97b4e6e",
    "build_date" : "2020-09-23T00:45:33.626720Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

stephenb · June 7, 2021, 2:19pm

log into the logstash container and try the same curl command to check elasticsearch but you will need to use the elasticsearch container name you defined in your docker compose for elasticsearch

Log into logstash container.

curl -u user:pw http://elasticsearch:9200

Akanksha_Pandey · June 7, 2021, 2:22pm

I ran the curl command curl -u user:pw http://elasticsearch:9200 inside logstash conatiner and it gave this error message-

curl: (7) Failed to connect to ::1: Cannot assign requested address

Akanksha_Pandey · June 7, 2021, 2:28pm

@stephenb should I add xpack.security.transport.ssl.enabled: true in elasticsearch.yml file?

Is the issue Lack of SSL causing the container to not accept traffic?

Please give your suggestions.

system · July 5, 2021, 2:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.