Hello. I am currently dealing with Kibana being unreachable. The page states "{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}".
I don't believe this is an authentication issue as Kibana was working earlier today. However, this 503 error has been occurring on and off for several hours and is now happening all the time. Checking the Kibana logs, I'm seeing this issue.
>{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["warning","plugins","licensing"],"pid":32624,"message":"License information could not be obtained from Elasticsearch due to {\"error\":{\"root_cause\":[{\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [<http_request>] would be [5074209616/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [0/0b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]\",\"bytes_wanted\":5074209616,\"bytes_limit\":5036520243,\"durability\":\"TRANSIENT\"}],\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [<http_request>] would be [5074209616/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [0/0b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]\",\"bytes_wanted\":5074209616,\"bytes_limit\":5036520243,\"durability\":\"TRANSIENT\"},\"status\":429} error"}
{"type":"response","@timestamp":"2023-01-27T17:33:54-05:00","tags":[],"pid":32624,"method":"post","statusCode":503,"req":{"url":"/internal/security/session","method":"post","headers":{"host":"log.purduefed.com","connection":"keep-alive","content-length":"0","sec-ch-ua":"\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"109\", \"Chromium\";v=\"109\"","content-type":"application/json","kbn-version":"7.16.3","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://log.purduefed.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://log.purduefed.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.50.0.145","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","referer":"https://log.purduefed.com/app/dev_tools"},"res":{"statusCode":503,"responseTime":71,"contentLength":86},"message":"POST /internal/security/session 503 71ms - 86.0B"}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","searchprofiler"],"pid":32624,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","painlessLab"],"pid":32624,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","snapshotRestore"],"pid":32624,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","transform"],"pid":32624,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","remoteClusters"],"pid":32624,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","indexLifecycleManagement"],"pid":32624,"message":"You cannot use index_lifecycle_management because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","rollup"],"pid":32624,"message":"You cannot use rollup because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":32624,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["warning","plugins","securitySolution"],"pid":32624,"message":"Unable to verify endpoint policies in line with license change: failed to fetch package policies: [parent] Data too large, data for [<http_request>] would be [5074210548/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [932/932b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=932/932b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [5074210548/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [932/932b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=932/932b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]"}
One thing I do know from my cluster settings is that "indices.breaker.total.limit" : "80%". I'm not sure how to remedy at this point so that I could at least access Kibana again. I've also tried turning off services such as Fleet and Metricbeat. I know the cluster is currently yellow due to a few unassigned and initializing shards that were getting stuck also due to the "Data too large" error.
Thank you for any help