Kibana session issues

Elastic Stack Kibana
1

Question relating to session timeout issues. We have been having this issue since 7.11 and are currently on 7.13.4. I have been making changes to the following. I got tired of it, and went over board with the following configs:

xpack.security.session.idleTimeout: "1w"
xpack.security.session.lifespan: "1w"
xpack.security.session.cleanupInterval: "1w"

Myself and the users have seen the issue regardless of having one or multiple tabs open to kibana. Also, at times when the sessions end we are in the middle of using a browser. Other times one browser shuts, but not all of them. Thanks in advance for any ideas.

Alex

2

Can you elaborate on what the issue is, it's not completely clear to me what is happening.

3

A user will be in kibana working in either discover, dashboards or the SIEM and then their session cuts out unexpectedly. They find themselves back on the login page. Sometimes we are able to hit back and get right back to what we were doing other times that doesn't work. Basically, we get logged out unexpectedly and without warning multiple times a day.

4

Was my description still unclear?

Thanks,
Alex

5

It's clear, but I am unable to help sorry. Hopefully someone else can chime in.

6

thanks we have been working with elastic consultants as well. unfortunately, not sure where else to turn at this point.

7

I don't think you're alone on this, I've experienced this same thing (and still do on 7.15.1). I haven't bothered opening a ticket with support about it yet. But from what I can tell, there is something definitely wrong with session management for the last few releases. In 7.15.1 though, you will at least now get a pop-up warning that your session will expire ~5 minutes. Which I don't think exists pre-7.15.x (might be in 7.14.x, but not 100% sure), which was part of the problem of "randomly" being logged out.

Note when I say:

there is something definitely wrong with session management

I believe that session lifespan is something that isn't refreshed with activity like some other session managers do. So even if you aren't idle, the session lifespan will eventually timeout.

I also think (but haven't looked into it further), that a user can potentially get multiple sessions, and they can each have their own lifespans, and when one ends, you get kicked out of the others, but the other sessions lifespans still exists, and it becomes an annoying circle.

8

Ya that is making sense to me. I have sent the diagnostics to support four or five times now and still haven't received anything actionable. We are on 7.13.4 at the moment and don't get the time out warning, which is nice.

I tend to agree with you in regards to the session management issue, as this isn't the only issue we are having with configs not being read properly in the elastic environment. We have another issue where we try to give kibana access to more resources via the configs. Even though we change the configs it doesn't actually get the extra ram. The only thing that seems to work is starting the service from the command line and not from systemctl. The issue is that we have a hard time replicating these issues in other environments.

Thanks for the input. I will update if we make any progress. I recently added an additional two kibana servers. The unexpected timeouts have gone away, but usage is way down due to the holiday so we will see what happens this week. Which will be interesting as this may prove it is somehow a resource issue rather than a session management issue if the problem goes away.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

10

As tidbits for future users who ended up here in their search. Sessions manifest from a fruition of dynamic parts, so knowing necessary data points is key but it still takes a fair amount of problem boxing to resolve. Above folks were doing a great job talking through, but making sure everyone has backing context for future investigations ...

Here's the summation of common problem boxing's that I've seen:

  • [encryption] Kibana sessions encrypt off xpack.security.encryptionKey. This auto-generates as a random string on node start up if not explicitly set. This implicates if multiple Kibana nodes are used w/auto-generated strings, sessions round-robining to Kibana nodes will HTTP 401 from incorrect/cross-wired decryption. HTTP 401's are interpretted as invalidated user sessions. To avoid this, manually set a 32+ character string, prevent in future via kibana#92654).
  • [architecture] As more edge-case but same spirit as [encryption], single Kibana URL should only point to a single Elasticsearch cluster. Sessions back up to .kibana_security_session_1 and the second Elasticsearch cluster won't have the necessary session info resulting in HTTP 401.
  • [browser-side] Cookies in general, and in Kibana's case, are shared between browser tabs. You authenticate multiple users to Kibana from a single computer browser tab via incognito or browser profiles which separate out the cookies. This explains why a single tab logout would expect to log out all other tabs. Some companies have privacy/security features which remove their cookies periodically which causes logouts. You test this by logging in with incognito which removes the effect of most browser extensions. If the security feature is computer wide, you receive permission to test on an unaffected computer (or change the effecting interval) to confirm.
  • [realms] External realms (e.g. SAML) can also induce session timeouts. To isolate the problem as Kibana, you'll test with a built-in/native user. If it is your external realm, check iDP-side settings or if your external realm is token based, make sure it isn't Elasticsearch timing out instead of Kibana by testing raising xpack.security.authc.token.timeout.
  • [node left] As a final note referencing back to [encryption], since Kibana sessions depend on an index sitting on an Elasticsearch node, if that node leaves the cluster all user sessions will be invalidated. Probably Kibana will start erring other things as well.
  • [degraded] If Kibana becomes degraded this may impact sessions. Checked via plugin "security" health, IIRC.

Random closing-loop from above tidbits

  • User sessions are not invalidated per username (e.g. an incognito session will outlive a normal browser session logout for the same username)
  • Confirming xpack.security.session.lifespan is an absolute timeout
  • When troubleshooting, consider temporarily expanding Cluster logging related to security/realms. You'll be looking for SESSION_EXPIRED. (It may also help to pull a cross-tab browser HAR (e.g. SAML tracers can do this for you) if you're able to forecast a session expiration to see if Kibana expires in response to external realm or if first session expired message comes from it.) If that doesn't kick out your answer, consider expanding Kibana logging related to logger:plugins.security.
  • If you want to go into the weeds about what's stored in the cookie/whatnot, you can read up here & here.

TLDR - first testing steps

  1. In incognito browser, see if native users are experiencing unexpected session invalidations. 70% of user setups I've seen it's an external/token realm experiencing issues.
  2. Check kibana.yml to make sure above settings are healthy ([OG description, encryption, architecture]). Probably 20% of remaining cases.
  3. For last 10%, I usually check above debug logs for variant communication/network issues (e.g. certification, mapping lookup, etc)

Hope that helps provide a base to work forward!

3 Likes