Unable to login to Kibana -- 401 error

Elastic Stack Kibana
1

Hi Team,

I'm facing the authentication issue on the kibana server when I'm trying to login with elastic user or Kibana user.

Error getting: "We couldn't log you in. Please try again."

Please find the whole setup and versions,
http:1
kibana-proxy:latest
kibana:latest
elasticsearch:8.9.1
logstash:latest

docker-compose --version
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend
docker-compose version 1.29.2, build unknown

Please find the images available,

REPOSITORY          TAG       IMAGE ID       CREATED         SIZE
app_logstash        latest    5a4a59d37a06   47 hours ago    808MB
app_kibana-proxy    latest    7a4640362088   47 hours ago    942MB
httpd               1         baf4caf2b8d2   47 hours ago    57.8MB
app_kibana          latest    5b9ae500fa93   13 months ago   886MB
elasticsearch       8.9.1     0a61551b3020   13 months ago   1.34GB
app_elasticsearch   latest    26361334901b   13 months ago   1.34GB

Please find the logs,
httpd logs:

137.58.0.203 - - [23/Sep/2024:07:16:44 +0000] "POST /kibana/internal/security/login HTTP/1.1" 401 73
137.58.0.203 - - [23/Sep/2024:07:16:48 +0000] "GET /kibana/64802/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.chunk.154.js HTTP/1.1" 200 665
[Mon Sep 23 07:17:05.229451 2024] [ssl:info] [pid 9:tid 140139283569464] [client 137.58.0.203:64499] AH01964: Connection to child 71 established (server elk:443)
[Mon Sep 23 07:17:05.230345 2024] [ssl:info] [pid 9:tid 140139283569464] [client 137.58.0.203:64499] AH02008: SSL library error 1 in handshake (server elk:443)
[Mon Sep 23 07:17:05.230366 2024] [ssl:info] [pid 9:tid 140139283569464] SSL Library Error: error:0A000416:SSL routines::sslv3 alert certificate unknown (SSL alert number 46)
[Mon Sep 23 07:17:05.230371 2024] [ssl:info] [pid 9:tid 140139283569464] [client 137.58.0.203:64499] AH01998: Connection closed to child 71 with abortive shutdown (server elk:443)
[Mon Sep 23 07:17:05.231899 2024] [ssl:info] [pid 92:tid 140139282291512] [client 137.58.0.203:64500] AH01964: Connection to child 207 established (server elk:443)
137.58.0.203 - - [23/Sep/2024:07:17:05 +0000] "GET /kibana/api/saved_objects_tagging/tags HTTP/1.1" 200 190
[Mon Sep 23 07:17:07.203472 2024] [ssl:info] [pid 8:tid 140139283426104] [client 137.58.0.203:64504] AH01964: Connection to child 8 established (server elk:443)
[Mon Sep 23 07:17:07.204313 2024] [ssl:info] [pid 8:tid 140139283426104] [client 137.58.0.203:64504] AH02008: SSL library error 1 in handshake (server elk:443)
[Mon Sep 23 07:17:07.204334 2024] [ssl:info] [pid 8:tid 140139283426104] SSL Library Error: error:0A000416:SSL routines::sslv3 alert certificate unknown (SSL alert number 46)
[Mon Sep 23 07:17:07.204339 2024] [ssl:info] [pid 8:tid 140139283426104] [client 137.58.0.203:64504] AH01998: Connection closed to child 8 with abortive shutdown (server elk:443)
[Mon Sep 23 07:17:07.206015 2024] [ssl:info] [pid 9:tid 140139283426104] [client 137.58.0.203:64505] AH01964: Connection to child 72 established (server elk:443)

kibana-proxy logs:

warnings.warn(
172.18.0.2 - - [23/Sep/2024 05:56:54] "POST / HTTP/1.1" 200 -
proxy        2024-09-23 06:54:08,614 INFO     Authentication was successful for user 'zvenrac'
/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'elasticsearch'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'elasticsearch'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
proxy        2024-09-23 06:54:08,764 ERROR    Could not create a new user with name zvenrac
172.18.0.2 - - [23/Sep/2024 06:54:08] "POST / HTTP/1.1" 429 -
172.18.0.2 - - [23/Sep/2024 06:55:18] "POST / HTTP/1.1" 500 -
proxy        2024-09-23 06:55:18,004 ERROR    Could not authenticate new user, something must be wrong with the request  {'providerType': 'basic', 'providerName': 'basic', 'currentURL': 'https://100.89.0.22/kibana/login?next=%2Fkibana%2F', 'params': {'username': 'elastic', 'password': '<elastic_pass>'}}
proxy        2024-09-23 07:16:45,085 INFO     Authentication was unsuccessful for user 'zvenrav'
172.18.0.2 - - [23/Sep/2024 07:16:45] "POST / HTTP/1.1" 401 -
172.18.0.2 - - [23/Sep/2024 07:17:07] "POST / HTTP/1.1" 500 -
proxy        2024-09-23 07:17:07,219 ERROR    Could not authenticate new user, something must be wrong with the request  {'providerType': 'basic', 'providerName': 'basic', 'currentURL': 'https://100.89.0.22/kibana/login?next=%2Fkibana%2F', 'params': {'username': 'elastic', 'password': '<elastic_pass>'}}

Kibana logs:

Caused by:
                export_exception: bulk [default_local] reports failures when exporting documents
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at Monitoring.bulk (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/monitoring.js:53:16)
    at sendBulkPayload (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/kibana_monitoring/lib/send_bulk_payload.js:19:10)
    at BulkUploader._onPayload (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/kibana_monitoring/bulk_uploader.js:161:12)
    at BulkUploader._fetchAndUpload (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/kibana_monitoring/bulk_uploader.js:150:9)
[2024-09-23T07:23:16.190+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] Unable to bulk upload the stats payload to the local cluster
[2024-09-23T07:23:26.191+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] ResponseError: export_exception
        Caused by:
                export_exception: bulk [default_local] reports failures when exporting documents
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at Monitoring.bulk (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/monitoring.js:53:16)
    at sendBulkPayload (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/kibana_monitoring/lib/send_bulk_payload.js:19:10)
    at BulkUploader._onPayload (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/kibana_monitoring/bulk_uploader.js:161:12)
    at BulkUploader._fetchAndUpload (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/kibana_monitoring/bulk_uploader.js:150:9)
[2024-09-23T07:23:26.191+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] Unable to bulk upload the stats payload to the local cluster
[2024-09-23T07:23:26.835+00:00][ERROR][plugins.taskManager] Failed to poll for work: TimeoutError: Request timed out
[2024-09-23T07:23:36.191+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] ResponseError: export_exception

elasticsearch logs:

{"@timestamp":"2024-09-23T07:24:29.488Z", "log.level": "WARN", "message":"unexpected error while indexing monitoring document", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][write][T#8]","log.logger":"org.elasticsearch.xpack.monitoring.exporter.local.LocalExporter","elasticsearch.cluster.uuid":"_bcR1d-pSwG-AlprvdSiow","elasticsearch.node.id":"sbfQv1DpRvKPGza7TX9aQQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster","error.type":"org.elasticsearch.xpack.monitoring.exporter.ExportException","error.message":"org.elasticsearch.cluster.block.ClusterBlockException: index [.monitoring-es-7-2024.09.23] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];","error.stack_trace":"org.elasticsearch.xpack.monitoring.exporter.ExportException: org.elasticsearch.cluster.block.ClusterBlockException: index [.monitoring-es-7-2024.09.23] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];\n\tat org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)\n\tat java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1006)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)\n\tat org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129)\n\tat org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:110)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:160)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:152)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:235)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:628)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:623)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1016)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:793)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction.runReroutePhase(TransportReplicationAction.java:227)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:222)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:85)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:152)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:620)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:616)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$handleIndexActionAuthorizationResult$17(AuthorizationService.java:596)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:173)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$21(AuthorizationService.java:848)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$23(AuthorizationService.java:871)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:173)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:397)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:173)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$22(AuthorizationService.java:868)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:377)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$24(AuthorizationService.java:865)\n\tat java.base/java.util.HashMap.forEach(HashMap.java:1429)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$25(AuthorizationService.java:857)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:273)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:193)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:96)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:71)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1060)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeBulkItems(AuthorizationService.java:756)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:588)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1014)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:980)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:397)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:173)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:273)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:193)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:96)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:71)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1060)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:377)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:173)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:197)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:205)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:94)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\n\tat org.elasticsearch.security@8.9.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:199)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:598)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:740)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:293)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:217)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: org.elasticsearch.cluster.block.ClusterBlockException: index [.monitoring-es-7-2024.09.23] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:221)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:199)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction.blockExceptions(TransportReplicationAction.java:309)\n\tat org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:787)\n\t... 110 more\n"}

/_cat/nodes?v
/_cluster/health?v
/_cat/allocation?v
/_xpack/security/_authenticate?pretty

curl: (52) Empty reply from server --> (same responce for all the urls)

image
image821×527 16.1 KB

Please let me know if any details required.

Thanks & regard,
Dinakar.

2

Hi @dinakar,

Welcome back! Can you confirm the versions of Kibana and Elasticsearch? Your versions should match and I want to check if that's the case as the tags suggest it might not be (assuming latest is 8.15.1):

I can think of a few things to check judging by the information:

  1. Can you check the username and password combination are correct. I see two different usernames aside form the default elastic in the logs, one authorized and one not:

172.18.0.2 - - [23/Sep/2024 05:56:54] "POST / HTTP/1.1" 200 -
proxy 2024-09-23 06:54:08,614 INFO Authentication was successful for user 'zvenrac'

proxy 2024-09-23 06:54:08,764 ERROR Could not create a new user with name zvenrac

172.18.0.2 - - [23/Sep/2024 06:55:18] "POST / HTTP/1.1" 500 -
proxy 2024-09-23 06:55:18,004 ERROR Could not authenticate new user, something must be wrong with the request {'providerType': 'basic', 'providerName': 'basic', 'currentURL': 'https://100.89.0.22/kibana/login?next=%2Fkibana%2F', 'params': {'username': 'elastic', 'password': '<elastic_pass>'}}
proxy 2024-09-23 07:16:45,085 INFO Authentication was unsuccessful for user 'zvenrav'

  1. I see you are using a proxy. Can you check your proxy setting is configured correctly and share your proxy is not blocking requests?

As an aside, I see you are receiving unverified HTTPS warnings, and I wonder if you need to check/ add certificate configuration.

InsecureRequestWarning: Unverified HTTPS request is being made to host 'elasticsearch'. Adding certificate verification is strongly advised. See: Advanced Usage - urllib3 2.2.3.dev38 documentation

Let us know!

3

Hi Carly,

Thanks a lot for your response.

kibana@7c36ff844208:~$ bin/kibana --version
8.9.1
elasticsearch@829f68072ffa:~$ elasticsearch --version
Version: 8.9.1, Build: docker/a813d015ef1826148d9d389bd1c0d781c6e349f0/2023-08-10T05:02:32.517455352Z, JVM: 20.0.2

Version of both Kibana & elastic is matching.

Ponit No.1:
elastic user is authentication with same password for the soapUI requests and curl command to the same URL.
And zvenrac is my LDAP user which is configured in kibana-proxy.
Point No.2:

kibana-proxy:
    restart: unless-stopped
    container_name: kibana-proxy
    build:
      context: kibana-proxy/
    environment:
      #- HTTP_TRUSTED_GROUP_HEADER=IDM-GRP
      #- HTTP_TRUSTED_USER_HEADER=SM_USER
      #- DEFAULT_ROLE=viewer
      #- IDM_ROLES=role1,role2
      - LDAP_URL=ldaps://ericsson.se:636
      - ELASTIC_PASSWORD=<elastic_password>
      - ELASTIC_URL=https://elasticsearch:9200
      - KIBANA_URL=http://kibana:5601/kibana
    networks:
      - elk

And for your information this is a new setup, eventhough it is working fine till today morning. we are facing issues soon after today logging in to the service.

Thanks & Regards,
Dinakar.