Kibana 7.10.0. licensing error

Elastic Stack Kibana
,
1

After upgrading the stack from version 7.8.0 to 7.10.0, a license error appeared in Kibana.

Full kibana.log

Снимок экрана 2020-11-17 в 17.18.59
Снимок экрана 2020-11-17 в 17.18.592424×728 38.8 KB

xpack settings were included. After the update, the config did not change, only deleted the Kibana system indexes.
parameter: xpack.security.enabled: true
however, an error pops up in Kibana:

Снимок экрана 2020-11-17 в 17.18.34
Снимок экрана 2020-11-17 в 17.18.34814×570 23.9 KB

GET _license

       {
      "license" : {
        "status" : "active",
        "uid" : "520gdz41-489d-4353-a347-c3152f6af699",
        "type" : "basic",
        "issue_date" : "2019-07-22T12:29:06.971Z",
        "issue_date_in_millis" : 1563798546971,
        "max_nodes" : 1000,
        "issued_to" : "preprodlogs",
        "issuer" : "elasticsearch",
        "start_date_in_millis" : -1
      }
    }

Please tell me what could be the problem.

2

Hi @ans_x,

Looking at your logs, I'm seeing the following:

{"type":"log","@timestamp":"2020-11-19T16:20:40Z","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_xpack]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_xpack]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""} error"}

This tells me that you might not have Kibana's credentials configured properly, so Kibana can't talk to Elasticsearch. Have you set both elasticsearch.username and elasticsearch.password within your kibana.yml?

3

Yes, these two parameters are written in the config. It is strange that when the kibana restarts, it works for a while without errors and sees ES, but later the connection disappears.

kibana.yaml

server.port: 5601
server.host: "0"
server.maxPayloadBytes: 1048576
elasticsearch.requestTimeout: 60000
elasticsearch.shardTimeout: 60000
#elasticsearch.startupTimeout: 20000
#elasticsearch.sniffOnStart: true
#elasticsearch.sniffInterval: 60000
elasticsearch.sniffOnConnectionFault: true
elasticsearch.hosts:
  - http://XXX:9200
  - http://XXX:9200
  - http://XXX:9200
elasticsearch.username: "elastic"
elasticsearch.password: "XXX"
xpack.reporting.encryptionKey: "something_32_least_at_characters"
xpack.security.enabled: true
xpack.security.encryptionKey: "something_at_least_32_characters"
kibana.index: ".kibana"
logging.dest: /usr/share/kibana/kibana.log

Could this be a kibana bug in 7.10.0?

4

This is pretty strange. Can you try to reduce elasticsearch.hosts to a single entry, and see if the problem still persists? I want to try to eliminate possibilities.

Also, to confirm: are you running a single Kibana instance, or multiple Kibana instances?

Based on the logs, it looks like Kibana ran fine for ~6 minutes before it lost the connection. Does that sound about right to you? Is it consistently lasting about that long, or is the timing sporadic?

Once you lose the connection, does it ever come back on its own, or do you have to restart to reconnect?

5

A kibana instance is one. The error time is floating, usually a couple of hours can work normally.

No recovery was observed. This is a test environment and it is rare.

6

If we can't isolate the problem to any one Elasticsearch node, then we'll probably need debug logs to triage this further, so that we can observe what's happening before and after these errors start appearing.

# kibana.yml
logging.verbose: true
7

I will attach a complete log from the start of the service with debug enabled with a link to Google Drive.
Presumably the problem started on 2020-11-20T18: 00

Full log by link in Google Drive

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.