How to solve - "We couldn't log you in. Please try again." - error in kibana

[M_S]

I tried to open kibana on my properly working ELK stack one fine morning only to discover this error message -

We couldn't log you in. Please try again.

I opened kibana.log to discover this following log entry coming repeatedly -

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T13:39:26.826+00:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1547619},"trace":{"id":"c43e53e94eb18e4bffa0558700a"},"transaction":{"id":"74231414395e"}}

Also, in elasticsearch.log we are getting some log entries also,

[2023-05-05T14:03:50,192][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elasticsearch01] high disk watermark [90%] exceeded on [KEUPBoHuRLKfQ][elasticsearch01][/var/lib/elasticsearch] free: 5.1gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete

Now, I deletd some indexes using curl operation but still can't login to kibana. What can we do to solve this problem?

[elasticforme (Sachin)]

is your cluster is healthy?

curl -u user:passwd -XGET server:9100/_cluster/healthy?pretty

if this works that means your user/password is working

may be restart kibana. may be disk is not below watermark ( which may be default which is 85%)

[M_S]

Hi, I solved the issue ! disk was not below watermark. Thank you for pinpointing the probable cause. However I used

curl -u elastic_instance:password -k -XDELETE "https://ip/index_name"

curl XDELETE to delete some of the biggest indices. Is there a more clean solution to this problem?

[elasticforme (Sachin)]

There is no other option. infect I just did some hard test.
if you ever fill up your disk 100% by something then you gone a have hard time getting this under control.

there are few ways trying like

1. getting disk monitor via cron bash.
2. checking via elastic monitoring
3. adding Xgig dummy file in disk. if anything goes wrong delete that file and you can atleast fix issue.
4. setting up something with delete logs file regular etc...