Kibana unreachable: 503 error due to circuit breaking exception

Hello. I am currently dealing with Kibana being unreachable. The page states "{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}".

I don't believe this is an authentication issue as Kibana was working earlier today. However, this 503 error has been occurring on and off for several hours and is now happening all the time. Checking the Kibana logs, I'm seeing this issue.

>{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["warning","plugins","licensing"],"pid":32624,"message":"License information could not be obtained from Elasticsearch due to {\"error\":{\"root_cause\":[{\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [<http_request>] would be [5074209616/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [0/0b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]\",\"bytes_wanted\":5074209616,\"bytes_limit\":5036520243,\"durability\":\"TRANSIENT\"}],\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [<http_request>] would be [5074209616/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [0/0b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]\",\"bytes_wanted\":5074209616,\"bytes_limit\":5036520243,\"durability\":\"TRANSIENT\"},\"status\":429} error"}
{"type":"response","@timestamp":"2023-01-27T17:33:54-05:00","tags":[],"pid":32624,"method":"post","statusCode":503,"req":{"url":"/internal/security/session","method":"post","headers":{"host":"log.purduefed.com","connection":"keep-alive","content-length":"0","sec-ch-ua":"\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"109\", \"Chromium\";v=\"109\"","content-type":"application/json","kbn-version":"7.16.3","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://log.purduefed.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://log.purduefed.com/app/dev_tools","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.50.0.145","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","referer":"https://log.purduefed.com/app/dev_tools"},"res":{"statusCode":503,"responseTime":71,"contentLength":86},"message":"POST /internal/security/session 503 71ms - 86.0B"}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","searchprofiler"],"pid":32624,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","painlessLab"],"pid":32624,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","snapshotRestore"],"pid":32624,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","transform"],"pid":32624,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","remoteClusters"],"pid":32624,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","indexLifecycleManagement"],"pid":32624,"message":"You cannot use index_lifecycle_management because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","rollup"],"pid":32624,"message":"You cannot use rollup because license information is not available at this time."}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":32624,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2023-01-27T17:33:54-05:00","tags":["warning","plugins","securitySolution"],"pid":32624,"message":"Unable to verify endpoint policies in line with license change: failed to fetch package policies: [parent] Data too large, data for [<http_request>] would be [5074210548/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [932/932b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=932/932b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [5074210548/4.7gb], which is larger than the limit of [5036520243/4.6gb], real usage: [5074209616/4.7gb], new bytes reserved: [932/932b], usages [request=58277888/55.5mb, fielddata=0/0b, in_flight_requests=932/932b, model_inference=0/0b, eql_sequence=3147824712/2.9gb, accounting=0/0b]"}

One thing I do know from my cluster settings is that "indices.breaker.total.limit" : "80%". I'm not sure how to remedy at this point so that I could at least access Kibana again. I've also tried turning off services such as Fleet and Metricbeat. I know the cluster is currently yellow due to a few unassigned and initializing shards that were getting stuck also due to the "Data too large" error.

Thank you for any help

Hi @mhoward

Have you made any progress?

There are a number of things that can cause this...

{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}"

Elaticsearch can be unavailable for a number of reasons...

Circuit Breakers
Generally, it is caused by being out of JVM Heap which can be from a number of things such as too many shards, fields, and mapping for the size/capacity of Node...

What version are you on?

What size is the JVM Heap for the data Nodes?

How many shards do you have on each node?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.