Kibana v7.16.1 login issue. Not able to login (ResponseError: version_conflict_engine_exception)

Elastic Stack Kibana
1

Elastic_login_error
Elastic_login_error962×757 28.6 KB

Hello,

We are on v7.16.1 of Kibana/Elastic and facing issues with Kibana login all of a sudden. It was working fine around couple of hours back When checked in the logs seeing below exceptions.

{"type":"log","@timestamp":"2022-07-01T12:01:16+00:00","tags":["error","plugins","security","session","index"],"pid":115029,"message":"Failed to create session value: version_conflict_engine_exception: [version_conflict_engine_exception] Reason: [F8GE4AZDdmPIGpePXKJ/T4fzvV2jwC5FQdz1N4mPseo=]: version conflict, document already exists (current version [1])"}
{"type":"log","@timestamp":"2022-07-01T12:01:16+00:00","tags":["error","http"],"pid":115029,"message":"ResponseError: version_conflict_engine_exception: [version_conflict_engine_exception] Reason: [F8GE4AZDdmPIGpePXKJ/T4fzvV2jwC5FQdz1N4mPseo=]: version conflict, document already exists (current version [1])\n at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:367:23)\n at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)\n at IncomingMessage.emit (node:events:402:35)\n at endReadableNT (node:internal/streams/readable:1343:12)\n at processTicksAndRejections (node:internal/process/task_queues:83:21) {\n meta: {\n body: { error: [Object], status: 409 },\n statusCode: 409,\n headers: {\n 'x-elastic-product': 'Elasticsearch',\n 'content-type': 'application/json; charset=UTF-8',\n 'content-length': '547'\n },\n meta: {\n context: null,\n request: [Object],\n name: 'elasticsearch-js',\n connection: [Object],\n attempts: 1,\n aborted: false\n }\n }\n}"}
{"type":"error","@timestamp":"2022-07-01T11:59:46+00:00","tags":,"pid":115029,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:61:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:172:34)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://<Kibana_Host>:<Kibana_Port>/internal/security/login","message":"Internal Server Error"}
{"type":"log","@timestamp":"2022-07-01T12:01:18+00:00","tags":["error","plugins","taskManager"],"pid":115029,"message":"Failed to poll for work: Error: work has timed out"}

Attached the login error screenshot as well. Did any one faced the similar issue and if yes, pleas post the solution to resolve this.

Cheers.

2

How are you creating your index pattern and did you also create an index pattern template?

@Pierre_Gayvallet how do we fix this?

Thanks,
Bhavya

3

@bhavyarm

Yes, We have created index templates and gave the index patterns to apply to that template. Also, We are using Data Streams instead of indices.

Any inputs on resolving this is much appreciated. Thanks.

4

@bhavyarm Any update on the issue?

5

@Avinash_09 - Did you happen to disable index.refresh_interval (i.e value = -1). We have seen similar errors in the past where the refresh interval was disabled on the .security-tokens-7 , .security-7 or .kibana_security_session_1 indices and this would result into similar issues. I would recommend to check these indices settings (c.f GET .kibana_security_session_1/_settings, etc.)

6

Hi @ropc ,

No, We haven't disabled any refresh interval settings on these indices in our environment. We have recently upgraded the Elastic stack to 8.3.3 to overcome this issue.

But, Post upgrade, When checked about this indices, For .kibana_security_session_1, .security-7 indices the refresh interval is with 1s and there is no index named .security-tokens-7. Getting below message in dev-tools:
'''
"type": "index_not_found_exception",
"reason": "no such index [.security-tokens-7]"
'''
What could be the next steps to resolve this issue? Can re-install of kibana nodes and joining them to existing elastic cluster would help? If yes, Should we need to follow any pre-steps before re-install of kibana to avoid any data loss w.r.t kibana related indices.

Regards,
Avinash

7

No, We haven't disabled any refresh interval settings on these indices in our environment. We have recently upgraded the Elastic stack to 8.3.3 to overcome this issue.

I am glad this is working after the upgrade.

there is no index named .security-tokens-7. Getting below message in dev-tools

The .security-tokens-7 index will only exists if you have authentication realms where access tokens need to be stored (e.g SAML authentication realm).

8

Hi @ropc,

Thanks for the quick reply. But, Post upgrade also we are still seeing the issue intermittently when accessing with kibana internal URL. Along with that, We are having new issue when accessing the Kibana via proxy/public URL.

Below is the error message:

Please upgrade your browser

This Elastic installation has strict security requirements enabled that your current browser does not meet

PFA screenshot for the same.

Kibana
Kibana1709×595 16.2 KB

Can you please help us here.

Regards,
Avinash

9

Post upgrade also we are still seeing the issue intermittently when accessing with kibana internal URL

Are you referring to the same error: version_conflict_engine_exception? Do you happen to have multiple Kibana instances running?

We are having new issue when accessing the Kibana via proxy/public URL.

It may be related to Content Security Policy (CSP) and/or some outdated browsers. You may want to check this.

10

Yes @ropc,

To your question 1: Yes, I am referring the same error. Yes, We have 2 kibana instances running for this cluster. Any other places to check on this issue?

Thanks for info on point 2. I will check on Content Security Policy (CSP) settings and see if the browser outdated issue gets resolved.

Regards,
Avinash

11

To your question 1: Yes, I am referring the same error. Yes, We have 2 kibana instances running for this cluster. Any other places to check on this issue?

Is this proxy used as a load-balancer as well? Perhaps you may want to check the following documentation:

12

Hi @ropc,

Yes, Proxy is used as load balancer. We have followed the necessary steps as per documentation. But, Will cross check on them and get back.

w.r.t browser issue (Content Security Policy), even after applying the settings still we see the same info message on browser.

Please upgrade your browser

This Elastic installation has strict security requirements enabled that your current browser does not meet.

PFA Screenshot for the same.

Kibana
Kibana1709×595 16.2 KB

Also, I could see below error journal logs. Is there anything to do with this error?

Aug 11 12:55:52 dev5083 kibana[97108]: [2022-08-11T12:55:52.888+00:00][ERROR][plugins.taskManager] Failed to poll for work: Error: work has timed out

13

  1. What is the current setting in Kibana for csp.strict and are you facing this issue on all browsers? Which browser / browser version are you using?

  2. Have you tried to bypass the proxy and check if the same problem happens?

14

csp.strict: true is the current setting in Kibana. Yes, I am facing issue on all browsers. I have tried on below browsers and their versions are mentioned as well.

Chrome Version: 104.0.5112.81 (Official Build) (64-bit)
Firefox Version: 103.0.2 (64-bit)
Microsoft Edge version: Version 104.0.1293.47 (Official build) (64-bit)

Yes, We tried bypassing the proxy and no problem is noticed when bypassing the proxy.

Regards,
Avinash

15

If bypassing the proxy works, then it must be something related to the proxy and its configuration. This is not really my expertise but I can ask around...

Which proxy are you using?

16

@ropc, Thanks for the update. We suspect the same. We are using Apache httpd as proxy server. Can you check and update us w.r.t the changes need to be done on Proxy server to fix this issue?

17

@Avinash_09 - One thing I can think about is the Apache httpd server overriding the Content-Security-Policy header. It is a wild guess but you may want to check with your IT team if any headers are being overridden at the proxy level.

18

Hi @ropc , Thank you very much for your time. We have made the changes at our proxy with our IT Team and now Public URL login issue is resolved.

Regards,
Avinash

1 Like
19

Hi @ropc ,

We are again now seeing the kibana login issue with below error. PFA below screenshot for the same

kibana-8.3.3_login_issue
kibana-8.3.3_login_issue1272×743 31.2 KB

Unable to find any related errors in kibana logs. Below is one of the error noticed in kibana logs:

{"type":"log","@timestamp":"2022-08-22T13:59:49+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":49237,"message":"TimeoutError: Request timed out\n at ClientRequest.onTimeout (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Connection.js:110:16)\n at ClientRequest.emit (node:events:526:28)\n at TLSSocket.emitRequestTimeout (node:_http_client:758:9)\n at Object.onceWrapper (node:events:645:28)\n at TLSSocket.emit (node:events:538:35)\n at TLSSocket.Socket._onTimeout (node:net:501:8)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7) {\n meta: {\n body: null,\n statusCode: null,\n headers: null,\n meta: {\n context: null,\n request: [Object],\n name: 'elasticsearch-js',\n connection: [Object],\n attempts: 0,\n aborted: false\n }\n },\n isBoom: true,\n isServer: true,\n data: null,\n output: {\n statusCode: 503,\n payload: {\n statusCode: 503,\n error: 'Service Unavailable',\n message: 'Request timed out'\n },\n headers: {}\n },\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'\n}"}

Not sure this is related to login issue or not.

Regards,
Avinash

20

@Avinash_09 - I am not sure as well - you would need to cross-reference the Kibana logs with the authentication issue. If this issue is reproducible, a HAR file capturing the issue along with the Kibana logs would usually help. However, it will be best if you can get this issue reviewed by our Support team (if you do have a subscription with us).