Kibana server is not ready yet

Elastic Stack Kibana
1

Hi,

I have upgrade my monolithic server using these guide: Upgrade Elasticsearch, Filebeat and Kibana - Upgrade guide

Now ,i have my server broken, kibana show always the message "kibana server is not ready yet" on screen.

I have seen filebeat service cannot index any event.
My elasticsearch state was red but 2h later gets yellow without any reason:

{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2974,
  "active_shards" : 2974,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 526,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 84.97142857142858
}

root@ip-172-24-16-180:/var/log# curl -X GET "http://localhost:9200/_cat/aliases?pretty"
auditbeat-7.12.1 auditbeat-7.12.1-2021.07.25-000004 - - - false
packetbeat-7.12.1 packetbeat-7.12.1-2021.05.21-000001 - - - false
packetbeat-7.13.4 packetbeat-7.13.4-2021.07.21-000001 - - - false
packetbeat-7.12.1 packetbeat-7.12.1-2021.05.25-000002 - - - false
packetbeat-7.13.4 packetbeat-7.13.4-2021.07.29-000005 - - - false
ilm-history-1 ilm-history-1-000015 - - - false
packetbeat-7.12.1 packetbeat-7.12.1-2021.08.23-000005 - - - true
filebeat-7.12.1 filebeat-7.12.1-2021.07.20-000003 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2021.08.03-000017 - - - true
metricbeat-7.6.2 metricbeat-7.6.2-2020.04.07-000001 - - - false
metricbeat-7.12.1 metricbeat-7.12.1-2021.07.20-000003 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.06.06-000003 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.12.03-000009 - - - false
packetbeat-7.12.1 packetbeat-7.12.1-2021.07.24-000004 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.07.06-000004 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2021.01.02-000010 - - - false
metricbeat-7.12.1 metricbeat-7.12.1-2021.06.20-000002 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2021.02.04-000011 - - - false
ilm-history-1 ilm-history-1-000016 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2021.05.05-000014 - - - false
packetbeat-7.13.4 packetbeat-7.13.4-2021.07.23-000002 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2021.07.04-000016 - - - false
packetbeat-7.13.4 packetbeat-7.13.4-2021.07.24-000003 - - - false
metricbeat-7.12.1 metricbeat-7.12.1-2021.05.21-000001 - - - false
packetbeat-7.13.4 packetbeat-7.13.4-2021.08.28-000006 - - - true
metricbeat-7.6.2 metricbeat-7.6.2-2021.04.05-000013 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2021.03.06-000012 - - - false
metricbeat-7.12.1 metricbeat-7.12.1-2021.08.19-000004 - - - true
metricbeat-7.6.2 metricbeat-7.6.2-2020.11.03-000008 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.09.04-000006 - - - false
packetbeat-7.13.4 packetbeat-7.13.4-2021.07.27-000004 - - - false
filebeat-7.12.1 filebeat-7.12.1-2021.08.19-000004 - - - true
auditbeat-7.12.1 auditbeat-7.12.1-2021.05.21-000001 - - - false
auditbeat-7.12.1 auditbeat-7.12.1-2021.08.17-000005 - - - true
.kibana_task_manager .kibana_task_manager_1 - - - -
ilm-history-1 ilm-history-1-000017 - - - true
filebeat-7.12.1 filebeat-7.12.1-2021.05.21-000001 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.10.04-000007 - - - false
auditbeat-7.12.1 auditbeat-7.12.1-2021.07.13-000003 - - - false
auditbeat-7.12.1 auditbeat-7.12.1-2021.06.20-000002 - - - false
filebeat-7.12.1 filebeat-7.12.1-2021.06.20-000002 - - - false
.kibana .kibana_2 - - - -
metricbeat-7.6.2 metricbeat-7.6.2-2021.06.04-000015 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.05.07-000002 - - - false
metricbeat-7.6.2 metricbeat-7.6.2-2020.08.05-000005 - - - false
packetbeat-7.12.1 packetbeat-7.12.1-2021.06.24-000003 - - - false
ilm-history-1 ilm-history-1-000014 - - - false

My install versions are:

Kibana:7.11.2
ELK:7.11.2
Filebeat: 7.11.2
Wazuh-manager: 4.2.0-1

thnx

2

To which version are you upgrading?

3

Hi,

Elasticsearch -> 7.11.2
Kibana -> 7.11.2
Filebeat -> 7.11.2
Wazuh-Manager -> 4.2.0-1 (from 3.12)

Thnx.

4

Welcome to our community! :smiley:

I wanted to add a note that you have way too many shards for your node. You should seriously consider deleting old data, or reindexing into monthly indices with a single shard. Otherwise you are just wasting resources and overloading Elasticsearch.

5

Hi,

The problem is... that i need to keep the data for complience. I need to track instantlly all one-year old data from my system. :frowning:

Thnx.

6

That's understandable. Look at using monthly indices rather than daily ones then.

You can easily run a reindex for that;

POST _reindex
{
  "source": {
    "index": "metricbeat-7.6.2-2020.08.*"
  },
  "dest": {
    "index": "metricbeat-7.6.2-2020.08"
  }
}

It looks like you are using ILM as well. What are the settings for that?

(I know these don't solve your immediate issue, but this is all leg work for getting Elasticsearch back to being in good health, which will help your Kibana issue.)

7

Hola:

Will I loose some type of data with the reindex?

I have not change any config for ILM. It is as default.
Where can i see these config?

Thnx.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.