hi all,
I'm new on kubernates and ELK and "i beg your pardon" if i wil lask something stupid,
i googled a while for a solution but i found nothing so i decide to ask help....
in our configuration on kubernates, we have kibana:7.9.3-oss3
from one day to another we can not acess kibana anymore and get this error:
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
checking the pod log i get this error:
{"type":"log","@timestamp":"2021-03-19T09:43:39Z","tags":["error","http"],"pid":7,"message":"{ [circuit_breaking_exception] [parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb], with { bytes_wanted=124271188 & bytes_limit=123273216 & durability=\"PERMANENT\" } :: {\"path\":\"/.kibana/_doc/config%3A7.9.3\",\"query\":{},\"statusCode\":429,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\\\",\\\"bytes_wanted\\\":124271188,\\\"bytes_limit\\\":123273216,\\\"durability\\\":\\\"PERMANENT\\\"}],\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\\\",\\\"bytes_wanted\\\":124271188,\\\"bytes_limit\\\":123273216,\\\"durability\\\":\\\"PERMANENT\\\"},\\\"status\\\":429}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 429,\n displayName: 'TooManyRequests',\n message:\n '[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]: [circuit_breaking_exception] [parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb], with { bytes_wanted=124271188 & bytes_limit=123273216 & durability=\"PERMANENT\" }',\n path: '/.kibana/_doc/config%3A7.9.3',\n query: {},\n body:\n { error:\n { root_cause: [Array],\n type: 'circuit_breaking_exception',\n reason:\n '[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]',\n bytes_wanted: 124271188,\n bytes_limit: 123273216,\n durability: 'PERMANENT' },\n status: 429 },\n statusCode: 429,\n response:\n '{\"error\":{\"root_cause\":[{\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\",\"bytes_wanted\":124271188,\"bytes_limit\":123273216,\"durability\":\"PERMANENT\"}],\"type\":\"circuit_breaking_exception\",\"reason\":\"[parent] Data too large, data for [indices:data/read/get[s]] would be [124271188/118.5mb], which is larger than the limit of [123273216/117.5mb], real usage: [124271032/118.5mb], new bytes reserved: [156/156b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=156/156b, accounting=3043568/2.9mb]\",\"bytes_wanted\":124271188,\"bytes_limit\":123273216,\"durability\":\"PERMANENT\"},\"status\":429}',\n toString: [Function],\n toJSON: [Function],\n isBoom: true,\n isServer: true,\n data: null,\n output:\n { statusCode: 500,\n payload:\n { message: 'An internal server error occurred',\n statusCode: 500,\n error: 'Internal Server Error' },\n headers: {} },\n reformat: [Function],\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/generalError' }"}
{"type":"error","@timestamp":"2021-03-19T09:43:39Z","tags":[],"pid":7,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/","path":"/","href":"/"},"message":"Internal Server Error"}
{"type":"response","@timestamp":"2021-03-19T09:43:39Z","tags":[],"pid":7,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"remURL","x-request-id":"cc7ebfe58c56f48b0ad7848946faddcf","x-real-ip":"84.253.147.38","x-forwarded-for":"84.253.147.38","x-forwarded-host":"remURL","x-forwarded-port":"80","x-forwarded-proto":"http","x-scheme":"http","pragma":"no-cache","cache-control":"no-cache","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7"},"remoteAddress":"100.96.8.61","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"},"res":{"statusCode":500,"responseTime":6,"contentLength":9},"message":"GET / 500 6ms - 9.0B"}
{"type":"response","@timestamp":"2021-03-19T09:43:39Z","tags":[],"pid":7,"method":"get","statusCode":404,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"remURL","x-request-id":"05a85ffd5951726ad1a5229122ca448f","x-real-ip":"84.253.147.38","x-forwarded-for":"84.253.147.38","x-forwarded-host":"remURL","x-forwarded-port":"80","x-forwarded-proto":"http","x-scheme":"http","pragma":"no-cache","cache-control":"no-cache","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://remURL/","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7"},"remoteAddress":"100.96.8.61","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","referer":"https://remURL/"},"res":{"statusCode":404,"responseTime":2,"contentLength":9},"message":"GET /favicon.ico 404 2ms - 9.0B"}
can someone point me to how solve this situation?
regrds
Salvatore