Hi everyone,
I have a strange problem with kibana, which I cannot figure out. It is similar to a number of topics here (e.g. https://discuss.elastic.co/t/http-error-401-on-api-licensing-info/281542, https://discuss.elastic.co/t/nginx-as-a-reverse-proxy-for-kibana-gives-401-for-31997-bundles-resources/244151), but I cannot find any working solution.
We employ Elasticsearch and kibana in our microservice architecture. We use a Spring Boot Zuul Server as reversed proxy and gateway server running behind an apache web server. All worked well with http-only connections and without additional Elasticsearch authentification (https only on the apache, authentication in the Gateway server). We used corresponding basePath and reqriteBasePath rules:
basePath: "/analytics/kibana"
rewriteBasePath: true
Due to emerging security discussions, we decided to secure the individual Spring Boot services and to enable TLS between them.
I also enabled Basic security and TLS between the nodes of the Elasticsearch cluster. I followed the guides Konfigurieren von SSL, TLS und HTTPS zum Absichern von Elasticsearch, Kibana, Beats und Logstash | Elastic Blog and https://www.elastic.co/guide/en/elasticsearch/reference/8.0/security-basic-setup.html (and the following pages). Everything seems to work as expected except the kibana login.
Whenever I try to login, I am caught in a redirect loop showing only the kibana loading icon. Looking at the network traffic in the browser, I see a 401 to /api/licensing/info immediately before the redirect and reload occurs. I also find this error in the kibana logs (these are the only logged errors):
{
"type": "response",
"@timestamp": <Timestamp>,
"tags": [],
"pid": <XXX>,
"method": "get",
"statusCode": 401,
"req": {
"url": "/api/licensing/info",
"method": "get",
"headers": {
"sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"97\", \"Chromium\";v=\"97\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Windows\"",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"sec-fetch-site": "none",
"sec-fetch-mode": "navigate",
"sec-fetch-user": "?1",
"sec-fetch-dest": "document",
"accept-encoding": "gzip, deflate, br",
"accept-language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
"x-forwarded-host": <my-host-name>,
"x-forwarded-proto": "https",
"x-forwarded-port": "443",
"x-forwarded-for": <IP-Address>,
"host": <internal IP>,
"connection": "Keep-Alive"
},
"remoteAddress": <Gateway-IP>,
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
},
"res": {
"statusCode": 401,
"responseTime": 32,
"contentLength": 66
},
"message": "GET /api/licensing/info 401 32ms - 66.0B"
}
Using curl, I see that the login with the elastic superuser and the kibana_system user seem to work properly. Also the call to the login-URL gives a 200 resonse:
{
"type": "response",
"@timestamp": "2022-03-07T10:21:24+01:00",
"tags": [],
"pid": 3035608,
"method": "get",
"statusCode": 200,
"req": {
"url": "/login?next=%2Fanalytics%2Fkibana%2F",
"method": "get",
"headers": {
"sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"97\", \"Chromium\";v=\"97\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Windows\"",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "navigate",
"sec-fetch-dest": "document",
"referer": <referrer address>,
"accept-encoding": "gzip, deflate, br",
"accept-language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
"x-forwarded-host": <server-address>,
"x-forwarded-proto": "https",
"x-forwarded-port": "443",
"x-forwarded-for": <server IP>,
"host": <kibana host>,
"connection": "Keep-Alive"
},
"remoteAddress": "192.168.176.227",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36",
"referer": <server IP>
},
"res": {
"statusCode": 200,
"responseTime": 27,
"contentLength": 145721
},
"message": "GET /login?next=%2Fanalytics%2Fkibana%2F 200 27ms - 142.3KB"
}
However, calling the errorous url, e.g. /aspi/licensing/info, gives a 401 as well:
curl -k https://kibana_system:<password>@<IP-Address of elasticsearch cluster>:<Port>/api/licensing/info
{
"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "action [indices:data/read/get] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [api], this action is granted by the index privileges [read,all]"
}
],
"type": "security_exception",
"reason": "action [indices:data/read/get] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [api], this action is granted by the index privileges [read,all]"
},
"status": 403
}
Removing the ssl setup for kibana resulted in no change. The error occurs with Chrome and Firefox as well as in the incognito mode.
It looks a liitle bit to me, as if the role settings for the kibana user are not set properly. Can this be the case? How can I test and correct it?
Thanks a lot for your help!
Cheers,
Eike