ELK security 401

[gmhafiz (Hafiz Shafruddin)]

I wanted to try out Alerting in Kibana so security needs to be enabled. However, both logstash and kibana isn't able to log into Elasticsearch

ELK version: 7.10.1

docker-compose.log.yml

version: '3.4'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: elasticsearch
    expose:
      - 9200
    ports:
      - "9200:9200"
    environment:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
      discovery.type: "single-node"
      node.store.allow_mmap: "false"
      xpack.security.enabled: "true"
      ELASTIC_USERNAME: "elastic"
      ELASTIC_PASSWORD: "208SBhMQIRynj206JJ73"

  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.1
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      ELASTIC_USERNAME: "kibana_system"
      ELASTIC_PASSWORD: "55tgVaDoSnEyPfNAwHzZ"
      discovery.type: "single-node"
      xpack.security.enabled: "true"
      xpack.security.encryptionKey: "fasfhasfy893rdn123o4238fn4523nf3fn532f5n2"
      xpack.encryptedSavedObjects.encryptionKey: "fasfhasfy893rdn123o4238fn4523nf3fn532f5n2"

  logstash-agent:
    image: logstash:7.10.1
    volumes:
      - ./log/logstash-agent:/etc/logstash
    environment:
      xpack.security.enabled: "true"
      ELASTIC_USERNAME: "logstash_system"
      ELASTIC_PASSWORD: "QU0Hq68nnBGpd02OJshB"
    command: logstash -f /etc/logstash/logstash.conf
    ports:
      - "12201:12201/udp"

  logstash-central:
    image: logstash:7.10.1
    volumes:
      - ./log/logstash-central:/etc/logstash
    environment:
      xpack.security.enabled: "true"
      ELASTIC_USERNAME: "logstash_system"
      ELASTIC_PASSWORD: "QU0Hq68nnBGpd02OJshB"
    command: logstash -f /etc/logstash/logstash.conf

  redis-cache:
    image: bitnami/redis:6.0

logstash-agent/logstash.yml

input {
  gelf {
    port => 12201
  }
}output {
  redis {
    host => "redis-cache"
    data_type => "list"
    key => "logstash"
  }
}

logstash-central/logstash.yml

input {
  redis {
    host => "redis-cache"
    type => "redis-input"
    data_type => "list"
    key => "logstash"
  }
}output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
  }
}

Run with

docker-compose -f docker-compose.log.yml up

I set up password with

docker exec -it elasticsearch bash
bin/elasticsearch-setup-passwords auto

Then copy the passwords in the yml file above.

Restart stack

ctrl+C
docker-compose -f docker-compose.log.yml up

I am able to log into Elasticsearch

curl -u elastic:208SBhMQIRynj206JJ73 http://localhost:9200/_security/_authenticate | jq

{
  "username": "elastic",
  "roles": [
    "superuser"
  ],
  "full_name": null,
  "email": null,
  "metadata": {
    "_reserved": true
  },
  "enabled": true,
  "authentication_realm": {
    "name": "reserved",
    "type": "reserved"
  },
  "lookup_realm": {
    "name": "reserved",
    "type": "reserved"
  },
  "authentication_type": "realm"
}

However, all kibana, kibana_system and logstash fail to login.
curl -u kibana_system:55tgVaDoSnEyPfNAwHzZ http://localhost:9200/_security/_authenticate | jq

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "unable to authenticate user [kibana_system] for REST request [/_security/_authenticate]",
        "header": {
          "WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type": "security_exception",
    "reason": "unable to authenticate user [kibana_system] for REST request [/_security/_authenticate]",
    "header": {
      "WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status": 401
}

docker logs kibana output

https://ghostbin.com/t2f65

[warkolm (Mark Walkom)]

Welcome to our community!
I would suggest using 7.15 as it's the latest release, 7.10 is nearly a year old. Also a 1GB heap for Elasticsearch is a little small, you'd be better off starting with 2GB if you can.

@gmhafiz

However, all kibana, kibana_system and logstash fail to login.

It might be worth updating one of those users in Elasticsearch with the password you have to make sure that it's correct.

[gmhafiz (Hafiz Shafruddin)]

Hi, I've updated to v7.15.0

Also, I've increased to 2G ES_JAVA_OPTS: "-Xms2g -Xmx2g"

I also updated the passwords to those users with

curl -X POST -u elastic:208SBhMQIRynj206JJ73 "localhost:9200/_security/user/logstash_system/_password?pretty" -H 'Content-Type: application/json' -d'
    {
    "password" : "xnyT25inZbyfeARUzcyK"
    }
    '

I updated the .yml file with the new passwords, then I restarted the containers. I am able to login with both kibana_system and logstash_system now

curl -u kibana_system:hW6Abxxw8eYSTysduvUT http://localhost:9200/_security/_authenticate | jq
{
  "username": "kibana_system",
  "roles": [
    "kibana_system"
  ],
  "full_name": null,
  "email": null,
  "metadata": {
    "_reserved": true
  },
  "enabled": true,
  "authentication_realm": {
    "name": "reserved",
    "type": "reserved"
  },
  "lookup_realm": {
    "name": "reserved",
    "type": "reserved"
  },
  "authentication_type": "realm"
}

# and
curl -u logstash_system:xnyT25inZbyfeARUzcyK http://localhost:9200/_security/_authenticate | jq

However, http://localhost:5601 is still inaccessible.

I get errors like

elasticsearch       | {"type": "server", "timestamp": "2021-09-29T05:57:48,997Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "ddb547b73f21", "message": "Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]" }
elasticsearch       | {"type": "server", "timestamp": "2021-09-29T05:57:49,601Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "ddb547b73f21", "message": "Active license is now [BASIC]; Security is enabled", "cluster.uuid": "NQhL9CS4S5OBfyXUmXptVA", "node.id": "WbMKUcL-S7eg0KSEAhyegw"  }

kibana              | {"type":"log","@timestamp":"2021-09-29T05:44:00+00:00","tags":["error","savedobjects-service"],"pid":1219,"message":"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.28.0.3:9200"}

logstash-agent_1    | [2021-09-29T05:57:49,350][WARN][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
logstash-central_1  | [2021-09-29T05:49:00,154][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}

I can still continue to login with both kibana_system and logstash_system though.

This is strange because it worked without security - I got my logs indexed properly and visualized in the kibana interface.