Hi all. I have been running ELK for a few months now version 7.12.0. I tried to log on today with username elastic and my browser saved password and it would not let me. I restarted the server but I cannot get to the log on screen now. I just get 'Kibana server is not ready yet' or sometimes
I have tried restarting both services (elasticsearch and kibana) both are running confirmed with
sudo service kibana status
sudo service elasticsearch status
The only thing I can think of is 95% of the disk is in use.
{"type":"log","@timestamp":"2021-07-26T12:04:18+00:00","tags":["info","plugins-system"],"pid":23806,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-26T12:04:18+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":23806,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-26T12:04:18+00:00","tags":["info","savedobjects-service"],"pid":23806,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-07-26T12:04:18+00:00","tags":["info","savedobjects-service"],"pid":23806,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE"}
{"type":"log","@timestamp":"2021-07-26T12:04:18+00:00","tags":["info","savedobjects-service"],"pid":23806,"message":"[.kibana_task_manager] Migration completed after 770ms"}
{"type":"log","@timestamp":"2021-07-26T12:04:47+00:00","tags":["warning","plugins","licensing"],"pid":23806,"message":"License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error"}
{"type":"log","@timestamp":"2021-07-26T12:04:48+00:00","tags":["warning","plugins-system"],"pid":23806,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
{"type":"log","@timestamp":"2021-07-26T12:11:20+00:00","tags":["warning","environment"],"pid":"25411","path":"/run/kibana/kibana.pid","message":"pid file already exists at /run/kibana/kibana.pid"}
root@kibana:/etc/elasticsearch#
{"type":"log","@timestamp":"2021-07-26T13:30:01+01:00","tags":["fatal","root"],"pid":4736,"message":"Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [cluster_block_exception]: index [.kibana_7.12.0_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];\n at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:139:13)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)\n at async Promise.all (index 0)\n at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:163:7)\n at Server.start (/usr/share/kibana/src/core/server/server.js:283:31)\n at Root.start (/usr/share/kibana/src/core/server/root/index.js:58:14)\n at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:100:5)\n at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:169:5)"}
{"type":"log","@timestamp":"2021-07-26T13:30:01+01:00","tags":["info","plugins-system"],"pid":4736,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-07-26T13:30:01+01:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":4736,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-07-26T13:30:01+01:00","tags":["info","savedobjects-service"],"pid":4736,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-07-26T13:30:02+01:00","tags":["info","savedobjects-service"],"pid":4736,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE"}
{"type":"log","@timestamp":"2021-07-26T13:30:02+01:00","tags":["info","savedobjects-service"],"pid":4736,"message":"[.kibana_task_manager] Migration completed after 677ms"}
What do I need to do next to diagnose why it isn't working?
