Error "Kibana server is not ready yet"

I had the exact same problem after upgrading all Elastic components in my Stack.
Since it is more a "test environment", the simplest solution for me was deleting all indicies older than 6.4.2

1 Like

I've been told that this https://github.com/elastic/kibana/pull/24873 will fix this downgrading issue. But for now, when you downgrade, you need to manually delete the space specific Kibana "application privileges", e.g.:

curl -u elastic:changeme -XDELETE localhost:9200/_xpack/security/privilege/kibana-.kibana/space_all
curl -u elastic:changeme -XDELETE localhost:9200/_xpack/security/privilege/kibana-.kibana/space_read
1 Like

I'm getting a similar No handler found for uri [/_xpack/security/privilege/kibana-.kibana/space_all] and method [DELETE] as above. I do have to connect to one ES node, right?

How would I - instead of downgrading - do the two steps listed in the Release notes?

Create a new role in Elasticsearch that has the all permission for the .tasks index
Create a new user in Elasticsearch that has the kibana_system role as well as the new role you just created

1 Like

I successfully downgraded by renaming kibana index. In kibana config changed from .kibana to .kibana2, then used below to move data from old index to new:
POST _reindex?wait_for_completion=true
{
"source": {
"index": ".kibana"
},
"dest": {
"index": ".kibana2"
}
}

1 Like

Ok, I was able to do the two steps using the Cerebro rest interface (I guess I'm doing something wrong with curl to get the No handler found error - I noticed in Cerebro that without the beginning / it works):

path: _xpack/security/role/tasks_bugfix

{
  "indices": [
    {
      "names": [ ".tasks" ],
      "privileges": [ "all" ]
    }
  ]
}

path: _xpack/security/user/user_bugfix

{
  "password" : "CHANGETHIS",
  "full_name" : "User Bugfix",
  "email" : "email@example.com",
  "roles" : [ "kibana_system", "tasks_bugfix" ]
}

Next I replaced elasticsearch.username and elasticsearch.password in /etc/kibana/kibana.yml with user_bugfix and CHANGETHIS, made sure that the two kibana_1 and kibana_2 indices were gone and started kibana.

Looks like it's working now.

Now I guess I need to wait for the bugfix release to remove the extra role and user and switch back to the old one.

1 Like

Hi,

Today also i updated kibana elasticsearch and logstash from 6.4 to 6.5 and i have exactly the same problem. Kibana cannot start and i am getting the same error "Kibana server is not ready yet"

Please can you help me to overcome this? How i can check the logs and what steps i have to do.
I have installed them all to a Centos 7 machine.

Best Regards,
Thanos

1 Like

I solved this problem by doing this

curl -XDELETE http://localhost:9200/.kibana

curl -XDELETE http://localhost:9200/.kibana*

curl -XDELETE http://localhost:9200/.kibana_2

curl -XDELETE http://localhost:9200/.kibana_1

9 Likes

The proper way is as described in the Release Notes

  1. Stop Kibana
    This depends on your operating system

  2. Delete the .kibana_1 and .kibana_2 indices that were created
    I used the Cerebro frontend to do that, because I keep having troubles with curl. But it should be possible to delete them using (localhost being an Elasticsearch node).

curl -XDELETE http://localhost:9200/.kibana_1
curl -XDELETE http://localhost:9200/.kibana_2
  1. Create a new role in Elasticsearch that has create_index , create , and read permissions for the .tasks index
    As I wrote above, curl didn't work, so I did this in the Cerebro rest interface:

path: _xpack/security/role/tasks_bugfix

{
  "indices": [
    {
      "names": [ ".tasks" ],
      "privileges": [ "all" ]
    }
  ]
}
  1. Create a new user in Elasticsearch that has the kibana_system role as well as the new role you just created (note: CHANGETHIS)

path: _xpack/security/user/user_bugfix

{
  "password" : "CHANGETHIS",
  "full_name" : "User Bugfix",
  "email" : "email@example.com",
  "roles" : [ "kibana_system", "tasks_bugfix" ]
}
  1. Update elasticsearch.username and elasticsearch.password in kibana.yml with the details from that new user
    The location of the kibana.yml depends on your installation

  2. If using a Kibana secure settings keystore, remove keys elasticsearch.username and elasticsearch.password from the keystore using the kibana-keystore tool. Add these keys back to the keystore using the new user and password as values.
    Since I don't use keystore yet, I can't comment on this, but the description seems good enough.

  3. Start Kibana This will be fixed in a future bug fix release, at which time you can go back to using the built-in kibana user.
    As above this depends on your system. After this bug is fixed in a later release you can undo all these changes.

2 Likes

Restarting all of the services fixed it for me.

sudo systemctl restart logstash elasticsearch kibana

And then if it still doesn't work, give Kibana one more restart and give it a little time.

sudo systemctl restart kibana

Check :slight_smile:

4 Likes

Is .kibana index real name? if it is alias name, index migration will fail.

1 Like

I just deleted both kibana_1 and kibana_2, but now my old index patterns, visualizations, everything is missing. Is there any way to get all that back?

2 Likes

This worked for me!!

Hi,

I have the same issue, did you have solde this problem ?

Regards,

Yes, but I solved it by going back to the version I had before. I tried to upgrade from 6.4.3 to 6.5.0. Luckily, after upgrading to 6.5.0 I saved all the older versions files in another folder, so at the end I solved it by running both older versions of elasticsearch and kibana (I don't use logstash).

So, in a few words, I decided not to run the new version for the moment.
I think it's better to use the snapshot and restore feature before upgrading (I didn't do it though).

Is this fixed in 6.5.1 / can we remove the addition role and user that we had to add for the temporary fix?

We've got 6.5.1 installed and I just ran into this issue. Balu, I'm working in an environment where I can't download Cerebro as of right now. Would you mind posting the curl commands you were using to see if mine are somewhat close to the right thing. It wouldnt surprise me if I have some syntax thats incorrect. I've been using variations of this to no availe:

curl -X POST -d {"indices":[{"names":[".tasks"],"privileges":["all"]}]} http://localhost:9200/_xpack/security/role/tasks_bugfix -H 'Content-Type: application/json'

Was the [globbing] nested braces not supported issue what you ran into?

I'm sorry, I didn't get the curl commands to work which is why I used Cerebro instead. The error messages I got was more about a wrong path than braces or similar though IIRC.

Have you tried single quotes around your "-d" option? I'd guess your shell is evaluating the braces...

I thought the problem itself was fixed in 6.5.1, so you shouldn't have run into it?

Delete all 4 index and a restart resolved my problem

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

For anyone visiting this topic, please see Kibana server is not ready yet.

2 Likes