Hello everyone,
First, let me say that I'm quite new to the ELK but I've tried my best to solve that issue myself.
I'm trying to migrate my ELK stack from docker-compose to kubernetes (so I'm not using elastic operators on kubernetes or elastic cloud, at least not yet).
I can't get Kibana server to be running. When using chrome to trying access kibana, I get message "Kibana server is not reaty yet".
When I check Kibana logs, I can see the following entries:
[2023-07-28T11:54:03.594+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-07-28T11:54:03.700+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-07-28T11:54:03.701+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-07-28T11:54:03.734+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 19ms.
[2023-07-28T11:54:03.738+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 19ms.
[2023-07-28T11:54:03.744+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [anonymous_user] with effective roles [metrics] on restricted indices [.kibana_task_manager_8.5.3_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 1 in 2 seconds.
[2023-07-28T11:54:03.744+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 6ms.
[2023-07-28T11:54:03.745+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [anonymous_user] with effective roles [metrics] on restricted indices [.kibana_8.5.3_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 1 in 2 seconds.
[2023-07-28T11:54:03.745+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 11ms.
[2023-07-28T11:54:04.292+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-07-28T11:54:05.751+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [anonymous_user] with effective roles [metrics] on restricted indices [.kibana_task_manager_8.5.3_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 2 in 4 seconds.
[2023-07-28T11:54:05.751+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 2007ms.
[2023-07-28T11:54:05.753+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [anonymous_user] with effective roles [metrics] on restricted indices [.kibana_8.5.3_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 2 in 4 seconds.
[2023-07-28T11:54:05.754+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 2008ms.
/usr/local/bin/import-patterns.sh;Waiting for kibana to come up...
/usr/local/bin/import-patterns.sh;OK - Kibana is up
My elasticsearch.yaml
elasticsearch.yml: |
---
## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0
node.name: "${SVC_ELASTIC}"
## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: basic
xpack.monitoring.collection.enabled: false
## Security
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/${SVC_ELASTIC}/${SVC_ELASTIC}.key"
xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/${SVC_ELASTIC}/${SVC_ELASTIC}.crt"
xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca/ca.crt"
xpack.security.authc:
anonymous:
username: anonymous_user
roles: metrics
authz_exception: true
# backup repo
path.repo: ["/backup"]
roles.yml: |
# The default roles file is empty as the preferred method of defining roles is
# through the API/UI. File based roles are useful in error scenarios when the
# API based roles may not be available
metrics:
cluster: ['monitor']
indices:
- names: [ '*' ]
privileges: [ 'read', 'view_index_metadata', 'manage', 'all' ]
My kibana.yaml
## Default Kibana configuration from Kibana base image.
## https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/templates/kibana_yml.template.js
#
server.name: tone-kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "${ELASTICSEARCH_HOSTS}" ]
monitoring.ui.container.elasticsearch.enabled: true
server.basePath: /app/kibana
# X-Pack security credentials
elasticsearch.username: "kibana_system"
elasticsearch.password: "${KIBANA_PASSWORD}"
elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/certs/ca/ca.crt"
I can normally curl the cluster from Kibana pod with elastic or kibana_system user.
Why I'm getting this error if 'metrics' role has 'manage' and 'all' privileges?
Why 'kibana_system' username is not used for that?