Endless loop while logging to Kibana with X-pack

Elastic Stack Kibana
1

Hi,

I have a elasticsearch v5 cluster of 10 nodes, running in docker swarm mode. All Elasticsearch containers are running the same image with X-pack installed. X-pack is installed in Kibana v5 as well, and Kibana is also running in docker.

However when I try to open kibana from a web browser a get this message in the browser:

The 10.10.10.91 page isn’t working
10.10.10.91 redirected you too many times.
Try clearing your cookies.
ERR_TOO_MANY_REDIRECT

I get this in Kibana's log:

    {"type":"log","@timestamp":"2016-12-01T07:59:59Z","tags":["security","warning"],"pid":12,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
    {"type":"log","@timestamp":"2016-12-01T07:59:59Z","tags":["status","plugin:console@5.0.1","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
    {"type":"log","@timestamp":"2016-12-01T08:00:00Z","tags":["status","plugin:timelion@5.0.1","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
    {"type":"log","@timestamp":"2016-12-01T08:00:00Z","tags":["listening","info"],"pid":12,"message":"Server running at http://0.0.0.0:5601"}
    {"type":"log","@timestamp":"2016-12-01T08:00:00Z","tags":["status","ui settings","error"],"pid":12,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
    {"type":"response","@timestamp":"2016-12-01T08:01:18Z","tags":[],"pid":12,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"10.10.10.91:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2915.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8,sv;q=0.6"},"remoteAddress":"10.255.0.2","userAgent":"10.255.0.2"},"res":{"statusCode":302,"responseTime":42,"contentLength":9},"message":"GET / 302 42ms - 9.0B"}
    {"type":"response","@timestamp":"2016-12-01T08:01:18Z","tags":[],"pid":12,"method":"get","statusCode":302,"req":{"url":"/login?next=%2F","method":"get","headers":{"host":"10.10.10.91:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2915.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8,sv;q=0.6"},"remoteAddress":"10.255.0.2","userAgent":"10.255.0.2"},"res":{"statusCode":302,"responseTime":6,"contentLength":9},"message":"GET /login?next=%2F 302 6ms - 9.0B"}

...
the last two lines of the log keeps repeating as the browser gets redirected

Doing a curl, shows the same:

$  curl -L -u elastic:changeme http://10.10.10.91:5601/ 
curl: (47) Maximum (50) redirects followed

Of course cleaning cookies and trying different browsers doesn't help.

Strangely enough, I got it to work, by restarting some of the Elasticsearch containers, but then it stopped working after a while. So I assume that KIbana is having problems with one of the Elasticsearch nodes. The ES nodes logs shows nothing unusual.

2

I don't know the cause of your problem yet, but here's another user with what looks like the same issue;

3

Is it possible you could download and install Kibana with X-Pack on another machine (your desktop/laptop, etc) to connect to your Elasticsearch nodes to see if that works OK (rule out a problem with the Elasticsearch nodes)?

4

Also, someone else reported this problem possibly related to CentOS kernel 3.10. And did not appear on later kernel version 3.19 through 4.8. Can you let us know what your OS/kernel version is?

Thanks,
Lee

5

I'm running Ubuntu with kernel 4 something. Tested from another machine as suggested and Kibana/x-pack showed the same behavior.

However I noticed in general that x-pack goes into the login loop when there is something wrong with the ELK cluster.

For example password set in kibana.yml for the kibana user, but default password not changed from changeme through the API.
Or if cluster is not formed due to too little nodes or some other errors, may cause this as well.
Also having too many red indices also led to the login loop. Deleted the red indices and kibana UI was able to load.

Therefore I assume that something with the ELK cluster is not right, which causes the login loop. However x-pack should print proper messages either in the logs or in the UI, rather than just looping the login screen.

I may be wrong, but this is my experience so far. Hope it helps someone.

1 Like
6

I just got word of another potential way to fix this problem.

Someone else reported;
" I got them working by having them set server.host in kibana.yml to the FQDN to match how their users are accessing Kibana (using FQDN and not IP) (instead of setting server.host to IP),"

Also, make sure elasticsearch is up and ready before starting kibana.

Regards,
Lee

7

I have server.host set to 0.0.0.0 and still the loop is gone when the ELK cluster is setup properly.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.