Kibana login Issue due to space full

Elastic Stack Kibana
1

Hello Expert,

We have created the Kibana and Elasticsearch with filebeat. Below are the details.
Kibana version: 7.14.1.
running in Kubernetes.

Issue:
Not able to login to Kibana as Elasticsearch space is full. but when i deleted the space and try to restart the pod at that time the Kibana pod is not coming up.

Kindly do the needful help as it making more trouble.

which indices should i delete or is there any other way to resolve this issue.

2

Welcome to our community! :smiley:

This is a really old version. You should be running 8.6 as the latest, but if you need to run 7 then use 7.17.

You will need to share more information for us to be able to help. Take a look at Dec 10th, 2022: [EN] Asking top notch technical questions to get you help quicker! and then update this topic with some extra info.

3

Thank you @warkolm for suggestion.
if we upgrade the version. but the disk full is going to come on that situation also. What are the precaution should we take to over come with this issue. Below is the error msg in logs we found.

logs of Kibana pod:
{"type":"log","@timestamp":"2023-03-21T09:32:00+00:00","tags":["info","plugins","security","routes"],"pid":1209,"message":"Logging in with provider "basic" (basic)"}
{"type":"log","@timestamp":"2023-03-21T09:32:01+00:00","tags":["error","plugins","security","session","index"],"pid":1209,"message":"Failed to create session value: cluster_block_exception: [cluster_block_exception] Reason: index [.kibana_security_session_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}
{"type":"log","@timestamp":"2023-03-21T09:32:01+00:00","tags":["error","http"],"pid":1209,"message":"ResponseError: cluster_block_exception: [cluster_block_exception] Reason: index [.kibana_security_session_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];\n at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:349:23)\n at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:275:11)\n at IncomingMessage.emit (events.js:412:35)\n at endReadableNT (internal/streams/readable.js:1317:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21) {\n meta: {\n body: { error: [Object], status: 429 },\n statusCode: 429,\n headers: {\n 'x-elastic-product': 'Elasticsearch',\n 'content-type': 'application/json; charset=UTF-8',\n 'content-length': '435'\n },\n meta: {\n context: null,\n request: [Object],\n name: 'elasticsearch-js',\n connection: [Object],\n attempts: 0,\n aborted: false\n }\n }\n}"}

4

Did you free up space on the Elasticsearch host?

5

Yes i tried manually cleaning and after we have restart the pod elastic and kibana pod. elastic pod came up but kibana pod is not coming up.

6

What did you do exactly?
What is the state of Elasticsearch? What does it respond with?

7

We use exec command to get in the kibana pod and going to the location where pvc volume is define getting in to the indices and deleting the old file.
after that the restart the pod and elastic pod came up. when i restart the elastic pod it not coming up giving below error.

this is the old pod
Action failed with 'search_phase_execution_exception: '. Retrying attempt 5 in 32 seconds."}
{"type":"log","@timestamp":"2023-03-23T05:08:09+00:00","tags":["info","savedobjects-service"],"pid":1211,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 16444ms."}

What is the state of Elasticsearch?
Elastic search is in red health status.

What does it respond with?
not bale to login 500 error

8

NEVER ever delete data directly from the disk of Elasticsearch.

You have probably severely broken Elasticsearch and your best bet would be to delete all data from the indices using the APIs and then starting over again.

9

How to delete the all the data with api as im not able to get it. Could you please help me with steps where to run that api and how to get info

10
11

Hi @warkolm

Thank you for shearing the link. im able to get the indices. which are holding spaced get with below command.

curl -X GET "localhost:9200/_cat/indices/my-index-*?v=true&s=index&pretty"

If i delete the index with curl command is it fine.

As i try to get the doc id but not able to get the doc id to delete. for below command.

curl -X DELETE "localhost:9200/my-index-000001/_doc/1?routing=shard-1&pretty"

12

Hello @warkolm

Thank you so much for guiding me. I'm able to logging after deleting the indices which consume more space.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.