X-Pack Monitoring screen error after ES upgrade

Elastic Stack Elasticsearch
1

Hello all,

I upgraded from previous (5.4.1) to latest release in order to be able to use .keyword fields in building a new advanced job in machine learning.

Since the upgrade, cluster monitoring screens give "Error 404 Not Found: undefined" error as attached.

Is this something to do with previous monitoring indices saved with old version?

1070×411 13.7 KB

Also, in syslog I found this repetitive message:

Unhandled rejection [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate="Basic realm="security" charset="UTF-8"" } } :: {"path":"/_xpac
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)
at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)
at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)

I changed users passwords again, as I had to reinstall x-packs in every cluster node, and reinstall kibana itself.

Any idea?

Thanks!
Leandro.

2

Maybe try clearing your browser history and cookies, and then load Monitoring by click the icon in the side bar.

The security_exception message looks pretty odd. Do you have the username and password for the Kibana user configured in kibana.yml? See elasticsearch.username and elasticsearch.password in https://www.elastic.co/guide/en/kibana/5.2/settings.html

3

Hi Tim,

Yes, double checked kibana user/pwd in kibana.yml, verified users have the needed roles and cleared browser history. No luck so far :expressionless:

Still giving "missing authentication token for REST request", even though users, roles, passwords are all in place. What gives? Is there any way to debug this?

Thanks!
Leandro.

4

You say you "reinstalled Kibana" after changing the passwords, but have you restarted Kibana after changing the password in the kibana.yml?

You can try stopping Kibana and they should stop. Then try navigating to http://localhost:9200/_xpack in a browser (replacing the URL for whatever URL Elasticsearch listens on), and try the username and password that you provided in the kibana.yml. You should see something like:

{
    "build": {
        "hash": "40710b2", 
        "date": "2017-05-29T16:24:24.726Z"
    }, 
    "license": {...}, 
    "features": {...}, 
    "tagline": "You know, for X"
}

Going back to this question, I'm still not entirely sure, but I was hoping we could figure out the authentication errors in the logs first. But note there were changes to the data schema from 5.4 to 5.5. Backwards compatibility should be maintained, but if you are having issues, you could try deleting the old monitoring data.

5

Hey Tim,

Yes, I see exactly that, using my authentication info with http://es-host:9200/_xpack. That's correct.

Unfortunately I cannot go through monitoring tab in kibana page, who gives always that "Monitoring: Error 404 Not Found: undefined" message.

Of course now the version is 5.5.1 and Build: 15405, but the message is still there.

In kibana.log message I see a repetitive message telling me that "License information could not be obtained from Elasticsearch. [security_exception] missing authentication token for REST request", even though I double checked kibana.yml settings and verified (I guess) user and role info in kibana.

What now? How can I further debug this? Is it really a licensing problem?

Thank you!
Leandro.

6

Do you happen to have a dedicated monitoring cluster configured in kibana.yml? That is, are you using the xpack.monitoring.elasticsearch.url setting?

If you are, make sure those settings are correct. If the dedicated monitoring cluster requires authorization, then you need to have xpack.monitoring.elasticsearch.username and xpack.monitoring.elasticsearch.password set as well.

7

Ok, Tim. Right on target!

I missed username and password for the (dedicated) monitoring cluster.

Now monitoring screens seem to be normal again.

Thank you!
Leandro.

8

Glad to hear it!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.