Kibana 500 ERROR (version 6.1.1)

claire_k · November 6, 2019, 1:27am

Hello,

I've got problem with Internal Server Error at KIBANA (6.1.1).
When I try to click 'Discover' tab, below error message showed.

Error: Internal Server Error
    at respond (http://127.0.0.1:5602/bundles/kibana.bundle.js?v=16350:61:161556)
    at checkRespForFailure (http://127.0.0.1:5602/bundles/kibana.bundle.js?v=16350:61:160796)
    at http://127.0.0.1:5602/bundles/kibana.bundle.js?v=16350:55:29074
    at processQueue (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:132456)
    at http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:133349
    at Scope.$digest (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:144239)
    at Scope.$apply (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:147018)
    at done (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:100026)
    at completeRequest (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:104697)
    at XMLHttpRequest.xhr.onload (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:105435)

And I can see below error too at Chrome developer mode.

Failed to load resource: the server responded with a status of 500 (Internal Server Error)
:5602/elasticsearch/_msearch:1

Also, I checked server error log.

Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/set_headers.js:15:11)
    at Object.mapUri (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/map_uri.js:59:53)
    at /data/elastic/kibana-6.1.1-linux-x86_64/node_modules/h2o2/lib/index.js:81:18
    at Object.internals.handler (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:96:36)
    at request._protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:30:23)
    at internals.Protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:64:5)
    at exports.execute (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:24:22)
    at each (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/request.js:384:16)
    at iterate (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:36:13)
    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:28:25)
    at Hoek.once (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:52:16)
    at wrapped (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:31:25)
    at Function.wrapped [as _next] (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
    at Function.internals.continue (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/reply.js:108:10)
    at /data/elastic/kibana-6.1.1-linux-x86_64/src/server/http/xsrf.js:25:26

I can't find the any solution for this issue.
Any suggestions on it?

mattkime · November 6, 2019, 4:03am

Hello @claire_k

Can you check your elasticsearch logs for errors?

claire_k · November 6, 2019, 4:36am

Hello, @mattkime
Thank you for your reply.

I've checked the log for elasticsearch but I can't find errors.
I think below logs look fine.

[2019-11-06T04:00:01,996][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [AAAAA01-2019.05.09/S_1P2DpqQW2e2WgeA2Gl1A] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [BBBBB01-2019.05.09/5v1ZaLLwSb2ipK4tKPPvmw] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [CCCCC01-2019.05.09/xARCaRHBS8asgZqZ1sg8oQ] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [DDDDD02-2019.05.09/tCcShLLeS7GaWwNoh-lMuA] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [BBBBB02-2019.05.09/MTVy_7FISgymjR-JO7-j8w] deleting index
[2019-11-06T09:00:05,521][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [AAAAA01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:00:05,697][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [AAAAA01-2019.11.06/I4rzky8ERomqpigRRsK-Ew] create_mapping [doc]
[2019-11-06T09:00:57,664][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [CCCCC01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:00:57,775][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [CCCCC01-2019.11.06/kru6_VofSE2q5npYtoAtjw] create_mapping [doc]
[2019-11-06T09:01:11,581][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [BBBBB02-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:01:11,691][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [BBBBB02-2019.11.06/6f94BeG_SR-rwnOHt0ccgA] create_mapping [doc]
[2019-11-06T09:04:24,437][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [BBBBB01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:04:24,556][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [BBBBB01-2019.11.06/v_ii13wWTbqtvz3OZCDumA] create_mapping [doc]
[2019-11-06T09:04:25,162][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [DDDDD02-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:04:25,287][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [DDDDD02-2019.11.06/fmuTN8-FQJ2x7jWBDIr9Aw] create_mapping [doc]
[2019-11-06T09:05:59,734][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [DDDDD01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:05:59,846][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [DDDDD01-2019.11.06/g_fnzjBFTCa4940yFQe06A] create_mapping [doc]

mattkime · November 6, 2019, 2:36pm

You're correct, I'm not seeing any elasticsearch errors. Can you post a larger sample of the kibana log? Particularly content surrounding errors.

claire_k · November 7, 2019, 4:40am

{"type":"response","@timestamp":"2019-11-07T00:01:04Z","tags":[],"pid":718,"method":"post","statusCode":200,"req":{"url":"/api/saved_objects/bulk_get","method":"post","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","content-length":"70","accept":"application/json, text/plain, */*","origin":"http://127.0.0.1:5602","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://127.0.0.1:5602/app/kibana","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/app/kibana"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /api/saved_objects/bulk_get 200 8ms - 9.0B"}
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"get","statusCode":304,"req":{"url":"/bundles/4b5a84aaf1c9485e060c503a0ff8cadb.woff2","method":"get","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","origin":"http://127.0.0.1:5602","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","accept":"*/*","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7","if-none-match":"\"574ea2698c03ae9477db2ea3baf460ee32f1a7ea\"","if-modified-since":"Sun, 17 Dec 2017 20:32:28 GMT"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /bundles/4b5a84aaf1c9485e060c503a0ff8cadb.woff2 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"get","statusCode":304,"req":{"url":"/ui/fonts/open_sans/open_sans_v13_latin_700.woff2","method":"get","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","origin":"http://127.0.0.1:5602","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","accept":"*/*","referer":"http://127.0.0.1:5602/app/kibana","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7","if-none-match":"\"578d7b5ff3816f7f666984e303364c81be0771d0\"","if-modified-since":"Sun, 17 Dec 2017 20:32:28 GMT"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/app/kibana"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /ui/fonts/open_sans/open_sans_v13_latin_700.woff2 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"get","statusCode":304,"req":{"url":"/bundles/448c34a56d699c29117adc64c43affeb.woff2","method":"get","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","origin":"http://127.0.0.1:5602","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","accept":"*/*","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7","if-none-match":"\"ca35b697d99cae4d1b60f2d60fcd37771987eb07\"","if-modified-since":"Sun, 17 Dec 2017 20:32:28 GMT"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350"},"res":{"statusCode":304,"responseTime":1,"contentLength":9},"message":"GET /bundles/448c34a56d699c29117adc64c43affeb.woff2 304 1ms - 9.0B"}
{"type":"error","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"level":"error","error":{"message":"Uncaught error: Expected originalHeaders to be an object, but object given","name":"Error","stack":"Error: Uncaught error: Expected originalHeaders to be an object, but object given\n    at setHeaders (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/set_headers.js:15:11)\n    at Object.mapUri (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/map_uri.js:59:53)\n    at /data/elastic/kibana-6.1.1-linux-x86_64/node_modules/h2o2/lib/index.js:81:18\n    at Object.internals.handler (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:96:36)\n    at request._protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:30:23)\n    at internals.Protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:64:5)\n    at exports.execute (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:24:22)\n    at each (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/request.js:384:16)\n    at iterate (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:36:13)\n    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:28:25)\n    at Hoek.once (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:52:16)\n    at wrapped (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)\n    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:31:25)\n    at Function.wrapped [as _next] (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)\n    at Function.internals.continue (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/reply.js:108:10)\n    at /data/elastic/kibana-6.1.1-linux-x86_64/src/server/http/xsrf.js:25:26"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/elasticsearch/_msearch","path":"/elasticsearch/_msearch","href":"/elasticsearch/_msearch"},"message":"Uncaught error: Expected originalHeaders to be an object, but object given"}
Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/set_headers.js:15:11)
    at Object.mapUri (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/map_uri.js:59:53)
    at /data/elastic/kibana-6.1.1-linux-x86_64/node_modules/h2o2/lib/index.js:81:18
    at Object.internals.handler (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:96:36)
    at request._protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:30:23)
    at internals.Protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:64:5)
    at exports.execute (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:24:22)

claire_k · November 7, 2019, 4:40am

at each (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/request.js:384:16)
at iterate (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:36:13)
at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:28:25)
at Hoek.once (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:52:16)
at wrapped (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:31:25)
at Function.wrapped [as _next] (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
at Function.internals.continue (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/reply.js:108:10)
at /data/elastic/kibana-6.1.1-linux-x86_64/src/server/http/xsrf.js:25:26
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"post","statusCode":500,"req":{"url":"/elasticsearch/_msearch","method":"post","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","content-length":"715","accept":"application/json, text/plain, */*","origin":"http://127.0.0.1:5602","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","content-type":"application/x-ndjson","referer":"http://127.0.0.1:5602/app/kibana","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/app/kibana"},"res":{"statusCode":500,"responseTime":9,"contentLength":9},"message":"POST /elasticsearch/_msearch 500 9ms - 9.0B"}

claire_k · November 7, 2019, 4:55am

Whenever this error happened, I restart KIBANA only.
After restarting KIBANA it works for almost a day, but the error occurs again.

Anyway, please check the logs for KIBANA. I've divided the logs because of body limits.
Is there any wrong?

mattkime · November 7, 2019, 11:17pm

How are you running kibana? I suspect thats only a portion of the logs. Perhaps stdout and stderr are going to different logs. You can also use https://gist.github.com/ to post larger portions of content.

claire_k · November 12, 2019, 12:40am

I'm running kibana by using nohup command and & for background running.

nohup kibana &

So, the logs I 've checked are from nohup.out file.
Here is the link for the logs.

https://gist.github.com/jh11425/a9bf3544faca7ba466264d11a8a79ffe

mattkime · November 12, 2019, 2:24am

Honestly, still mysterious to me, but I'll keep plugging away.

I found this PR which modifies the affected behavior. Only useful for locating the error message - https://github.com/elastic/kibana/pull/28344/files/536f7d449ee32e3564bdda0be765b1b39c939b93#

How are you running Kibana and elasticsearch? Are either of them behind a proxy? Is there anything that might be modifying http headers? Can you share your kibana.yml and elasticsearch.yml content?

Akhil_K · November 18, 2019, 10:52am

Even I am facing this issue @claire_k . I get the below error message in UI which is totally confusing :

More details on UI side mentioned in below url :
The aggregations key is missing from the response, check your permissions for this request

I have been searching for some inputs for somedays but could not find any valid leads.

system · December 16, 2019, 11:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How I can fix {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"} in kibana Kibana	2	2568	November 10, 2017
Discover: internal server error Kibana	16	10900	September 9, 2019
Getting {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."} when accessing kibana dashboard Kibana	1	184	March 13, 2024
Discover page error in kibana Kibana	8	956	November 18, 2020
Kibana throwing 500 Internal Server Error Kibana	7	24228	August 3, 2017

Kibana 500 ERROR (version 6.1.1)

Related topics