Kibana get 502 Bad Gateway

Elastic Stack Kibana
1

Kibana status is active, but the webpage shows me 502 Bad Gateway. I checked the kibana.log. Here are the error message:

{"type":"log","@timestamp":"2022-05-17T18:56:18+00:00","tags":["error","savedobjects-service"],"pid":25479,"message":"[.kibana_task_manager] Unexpected Elasticsearch ResponseError: statusCode: 429, method: PUT, url: /.kibana_task_manager_7.14.2_001/_mapping?timeout=60s error: [cluster_block_exception]: index [.kibana_task_manager_7.14.2_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];,"}
{"type":"log","@timestamp":"2022-05-17T18:56:18+00:00","tags":["error","savedobjects-service"],"pid":25479,"message":"[.kibana_task_manager] migration failed, dumping execution log:"}
{"type":"log","@timestamp":"2022-05-17T18:56:18+00:00","tags":["fatal","root"],"pid":25479,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 429, method: PUT, url: /.kibana_task_manager_7.14.2_001/_mapping?timeout=60s error: [cluster_block_exception]: index [.kibana_task_manager_7.14.2_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];,\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:172:13)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async Promise.all (index 1)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:164:7)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:274:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:55:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:98:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:224:5)"}

The cluster health of Elasticsearch is good.

curl -X GET -k -u elastic https://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "Elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 201,
  "active_shards" : 402,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

How shall I fix this issue? Thanks.

2

I'd check the disk use on your Elasticsearch node.

3

I installed two nodes. Here are the df -h results of the two nodes:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  660K  797M   1% /run
/dev/sda1        91G   46G   41G  54% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           798M     0  798M   0% /run/user/1001

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  8.7M  789M   2% /run
/dev/sda1        91G   43G   44G  50% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           798M     0  798M   0% /run/user/1001

The disk usage is not big, so I am confused why it gave me disk space usage exceeding error.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.