Kibana : internal server error

Elastic Stack Kibana
1

hi all,

I'm new on kubernates and ELK and "i beg your pardon" if i wil lask something stupid,

i googled a while for a solution but i found nothing so i decide to ask help....

in our configuration on kubernates, we have kibana:7.9.3-oss3

from one day to another we can not acess kibana anymore and get this error:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

checking the pod log i get this error:

{"type":"log","@timestamp":"2021-03-19T09:43:39Z","tags":["error","http"],"pid":7,"message":"{ [circuit_breaking_exception] [parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb], with { bytes_wanted=124271188 & bytes_limit=123273216 & durability=\"PERMANENT\" } :: {\"path\":\"/.kibana/_doc/config%3A7.9.3\",\"query\":{},\"statusCode\":429,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\\\",\\\"bytes_wanted\\\":124271188,\\\"bytes_limit\\\":123273216,\\\"durability\\\":\\\"PERMANENT\\\"}],\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\\\",\\\"bytes_wanted\\\":124271188,\\\"bytes_limit\\\":123273216,\\\"durability\\\":\\\"PERMANENT\\\"},\\\"status\\\":429}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 429,\n  displayName: 'TooManyRequests',\n  message:\n   '[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]: [circuit_breaking_exception] [parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb], with { bytes_wanted=124271188 & bytes_limit=123273216 & durability=\"PERMANENT\" }',\n  path: '/.kibana/_doc/config%3A7.9.3',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'circuit_breaking_exception',\n        reason:\n         '[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]',\n        bytes_wanted: 124271188,\n        bytes_limit: 123273216,\n        durability: 'PERMANENT' },\n     status: 429 },\n  statusCode: 429,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\",\"bytes_wanted\":124271188,\"bytes_limit\":123273216,\"durability\":\"PERMANENT\"}],\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\",\"bytes_wanted\":124271188,\"bytes_limit\":123273216,\"durability\":\"PERMANENT\"},\"status\":429}',\n  toString: [Function],\n  toJSON: [Function],\n  isBoom: true,\n  isServer: true,\n  data: null,\n  output:\n   { statusCode: 500,\n     payload:\n      { message: 'An internal server error occurred',\n        statusCode: 500,\n        error: 'Internal Server Error' },\n     headers: {} },\n  reformat: [Function],\n  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/generalError' }"}
{"type":"error","@timestamp":"2021-03-19T09:43:39Z","tags":[],"pid":7,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/","path":"/","href":"/"},"message":"Internal Server Error"}
{"type":"response","@timestamp":"2021-03-19T09:43:39Z","tags":[],"pid":7,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"remURL","x-request-id":"cc7ebfe58c56f48b0ad7848946faddcf","x-real-ip":"84.253.147.38","x-forwarded-for":"84.253.147.38","x-forwarded-host":"remURL","x-forwarded-port":"80","x-forwarded-proto":"http","x-scheme":"http","pragma":"no-cache","cache-control":"no-cache","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7"},"remoteAddress":"100.96.8.61","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"},"res":{"statusCode":500,"responseTime":6,"contentLength":9},"message":"GET / 500 6ms - 9.0B"}
{"type":"response","@timestamp":"2021-03-19T09:43:39Z","tags":[],"pid":7,"method":"get","statusCode":404,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"remURL","x-request-id":"05a85ffd5951726ad1a5229122ca448f","x-real-ip":"84.253.147.38","x-forwarded-for":"84.253.147.38","x-forwarded-host":"remURL","x-forwarded-port":"80","x-forwarded-proto":"http","x-scheme":"http","pragma":"no-cache","cache-control":"no-cache","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://remURL/","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7"},"remoteAddress":"100.96.8.61","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://remURL/"},"res":{"statusCode":404,"responseTime":2,"contentLength":9},"message":"GET /favicon.ico 404 2ms - 9.0B"}

can someone point me to how solve this situation?

regrds
Salvatore

2

Hey there! From that top line in the error log you pasted, it's looking like you're hitting a circuit breaker that prevents the instance from running into an out of memory exception.

Looks like someone on the forums hit something similar to this: What does this error mean - Data too large, data for [<transport_request>]
Here's more info on the circuit breaker if you're curious: Improve Elasticsearch resiliency with the real memory circuit breaker | Elastic Blog

It seems like a good fix here would be to increase the amount of heap memory that your kibana instance is using so you're able to retrieve that data

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.