Ran across a minor issue logging in as I was getting the dreaded "all shards failed" error when accessing our QA environment via SAML. I tried to login directly by going to each node specific url where I ran into a different error on 2 out of our 3 nodes. Seems like node 1 was throwing the "all shards failed" error message and the other two were giving me the
{"statusCode":403,"error":"Forbidden","message":"Sorry, you already have an active Kibana session. If you want to start a new one, please logout from the existing session first."}
error message. Clearly my SAML sessions were not logging out for some reason. After reviewing the thread, Kibana logout not working, it seems like the most likely cause was that our sp.logout field was not setup correctly. This proved to be true as all of our nodes had https://kibana.gov in the field and not https://kibana.gov/logout, in essence I was never really logging out. Restart of all of the nodes fixed the problem and the cluster status went back to green after the shards allocated properly.
If anyone could explain why not being logged out of the SAML session may have caused the sharding error that would be appreciated. Just for some insight into how things work, if of course, my assumption that not being logged out was causing the issue.