Kibana 500 ERROR (version 6.1.1)

Elastic Stack Kibana
1

Hello,

I've got problem with Internal Server Error at KIBANA (6.1.1).
When I try to click 'Discover' tab, below error message showed.

Error: Internal Server Error
    at respond (http://127.0.0.1:5602/bundles/kibana.bundle.js?v=16350:61:161556)
    at checkRespForFailure (http://127.0.0.1:5602/bundles/kibana.bundle.js?v=16350:61:160796)
    at http://127.0.0.1:5602/bundles/kibana.bundle.js?v=16350:55:29074
    at processQueue (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:132456)
    at http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:133349
    at Scope.$digest (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:144239)
    at Scope.$apply (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:147018)
    at done (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:100026)
    at completeRequest (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:104697)
    at XMLHttpRequest.xhr.onload (http://127.0.0.1:5602/bundles/commons.bundle.js?v=16350:29:105435)

And I can see below error too at Chrome developer mode.

Failed to load resource: the server responded with a status of 500 (Internal Server Error)
:5602/elasticsearch/_msearch:1

Also, I checked server error log.

Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/set_headers.js:15:11)
    at Object.mapUri (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/map_uri.js:59:53)
    at /data/elastic/kibana-6.1.1-linux-x86_64/node_modules/h2o2/lib/index.js:81:18
    at Object.internals.handler (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:96:36)
    at request._protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:30:23)
    at internals.Protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:64:5)
    at exports.execute (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:24:22)
    at each (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/request.js:384:16)
    at iterate (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:36:13)
    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:28:25)
    at Hoek.once (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:52:16)
    at wrapped (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:31:25)
    at Function.wrapped [as _next] (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
    at Function.internals.continue (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/reply.js:108:10)
    at /data/elastic/kibana-6.1.1-linux-x86_64/src/server/http/xsrf.js:25:26

I can't find the any solution for this issue.
Any suggestions on it?

2

Hello @claire_k

Can you check your elasticsearch logs for errors?

3

Hello, @mattkime
Thank you for your reply.

I've checked the log for elasticsearch but I can't find errors.
I think below logs look fine.

[2019-11-06T04:00:01,996][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [AAAAA01-2019.05.09/S_1P2DpqQW2e2WgeA2Gl1A] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [BBBBB01-2019.05.09/5v1ZaLLwSb2ipK4tKPPvmw] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [CCCCC01-2019.05.09/xARCaRHBS8asgZqZ1sg8oQ] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [DDDDD02-2019.05.09/tCcShLLeS7GaWwNoh-lMuA] deleting index
[2019-11-06T04:00:02,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [QkyvuFa] [BBBBB02-2019.05.09/MTVy_7FISgymjR-JO7-j8w] deleting index
[2019-11-06T09:00:05,521][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [AAAAA01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:00:05,697][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [AAAAA01-2019.11.06/I4rzky8ERomqpigRRsK-Ew] create_mapping [doc]
[2019-11-06T09:00:57,664][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [CCCCC01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:00:57,775][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [CCCCC01-2019.11.06/kru6_VofSE2q5npYtoAtjw] create_mapping [doc]
[2019-11-06T09:01:11,581][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [BBBBB02-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:01:11,691][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [BBBBB02-2019.11.06/6f94BeG_SR-rwnOHt0ccgA] create_mapping [doc]
[2019-11-06T09:04:24,437][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [BBBBB01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:04:24,556][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [BBBBB01-2019.11.06/v_ii13wWTbqtvz3OZCDumA] create_mapping [doc]
[2019-11-06T09:04:25,162][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [DDDDD02-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:04:25,287][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [DDDDD02-2019.11.06/fmuTN8-FQJ2x7jWBDIr9Aw] create_mapping [doc]
[2019-11-06T09:05:59,734][INFO ][o.e.c.m.MetaDataCreateIndexService] [QkyvuFa] [DDDDD01-2019.11.06] creating index, cause [auto(bulk api)], templates [basic], shards [1]/[1], mappings []
[2019-11-06T09:05:59,846][INFO ][o.e.c.m.MetaDataMappingService] [QkyvuFa] [DDDDD01-2019.11.06/g_fnzjBFTCa4940yFQe06A] create_mapping [doc]
4

You're correct, I'm not seeing any elasticsearch errors. Can you post a larger sample of the kibana log? Particularly content surrounding errors.

11 
{"type":"response","@timestamp":"2019-11-07T00:01:04Z","tags":[],"pid":718,"method":"post","statusCode":200,"req":{"url":"/api/saved_objects/bulk_get","method":"post","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","content-length":"70","accept":"application/json, text/plain, */*","origin":"http://127.0.0.1:5602","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://127.0.0.1:5602/app/kibana","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/app/kibana"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /api/saved_objects/bulk_get 200 8ms - 9.0B"}
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"get","statusCode":304,"req":{"url":"/bundles/4b5a84aaf1c9485e060c503a0ff8cadb.woff2","method":"get","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","origin":"http://127.0.0.1:5602","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","accept":"*/*","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7","if-none-match":"\"574ea2698c03ae9477db2ea3baf460ee32f1a7ea\"","if-modified-since":"Sun, 17 Dec 2017 20:32:28 GMT"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /bundles/4b5a84aaf1c9485e060c503a0ff8cadb.woff2 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"get","statusCode":304,"req":{"url":"/ui/fonts/open_sans/open_sans_v13_latin_700.woff2","method":"get","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","origin":"http://127.0.0.1:5602","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","accept":"*/*","referer":"http://127.0.0.1:5602/app/kibana","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7","if-none-match":"\"578d7b5ff3816f7f666984e303364c81be0771d0\"","if-modified-since":"Sun, 17 Dec 2017 20:32:28 GMT"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/app/kibana"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /ui/fonts/open_sans/open_sans_v13_latin_700.woff2 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"get","statusCode":304,"req":{"url":"/bundles/448c34a56d699c29117adc64c43affeb.woff2","method":"get","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","origin":"http://127.0.0.1:5602","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","accept":"*/*","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7","if-none-match":"\"ca35b697d99cae4d1b60f2d60fcd37771987eb07\"","if-modified-since":"Sun, 17 Dec 2017 20:32:28 GMT"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/bundles/commons.style.css?v=16350"},"res":{"statusCode":304,"responseTime":1,"contentLength":9},"message":"GET /bundles/448c34a56d699c29117adc64c43affeb.woff2 304 1ms - 9.0B"}
{"type":"error","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"level":"error","error":{"message":"Uncaught error: Expected originalHeaders to be an object, but object given","name":"Error","stack":"Error: Uncaught error: Expected originalHeaders to be an object, but object given\n    at setHeaders (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/set_headers.js:15:11)\n    at Object.mapUri (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/map_uri.js:59:53)\n    at /data/elastic/kibana-6.1.1-linux-x86_64/node_modules/h2o2/lib/index.js:81:18\n    at Object.internals.handler (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:96:36)\n    at request._protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:30:23)\n    at internals.Protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:64:5)\n    at exports.execute (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:24:22)\n    at each (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/request.js:384:16)\n    at iterate (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:36:13)\n    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:28:25)\n    at Hoek.once (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:52:16)\n    at wrapped (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)\n    at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:31:25)\n    at Function.wrapped [as _next] (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)\n    at Function.internals.continue (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/reply.js:108:10)\n    at /data/elastic/kibana-6.1.1-linux-x86_64/src/server/http/xsrf.js:25:26"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/elasticsearch/_msearch","path":"/elasticsearch/_msearch","href":"/elasticsearch/_msearch"},"message":"Uncaught error: Expected originalHeaders to be an object, but object given"}
Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/set_headers.js:15:11)
    at Object.mapUri (/data/elastic/kibana-6.1.1-linux-x86_64/src/core_plugins/elasticsearch/lib/map_uri.js:59:53)
    at /data/elastic/kibana-6.1.1-linux-x86_64/node_modules/h2o2/lib/index.js:81:18
    at Object.internals.handler (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:96:36)
    at request._protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:30:23)
    at internals.Protect.run (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:64:5)
    at exports.execute (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/handler.js:24:22)
12 
at each (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/request.js:384:16)
at iterate (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:36:13)
at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:28:25)
at Hoek.once (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/protect.js:52:16)
at wrapped (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
at done (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/node_modules/items/lib/index.js:31:25)
at Function.wrapped [as _next] (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hoek/lib/index.js:875:20)
at Function.internals.continue (/data/elastic/kibana-6.1.1-linux-x86_64/node_modules/hapi/lib/reply.js:108:10)
at /data/elastic/kibana-6.1.1-linux-x86_64/src/server/http/xsrf.js:25:26
{"type":"response","@timestamp":"2019-11-07T00:01:05Z","tags":[],"pid":718,"method":"post","statusCode":500,"req":{"url":"/elasticsearch/_msearch","method":"post","headers":{"host":"127.0.0.1:5602","connection":"keep-alive","content-length":"715","accept":"application/json, text/plain, */*","origin":"http://127.0.0.1:5602","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36","content-type":"application/x-ndjson","referer":"http://127.0.0.1:5602/app/kibana","accept-encoding":"gzip, deflate","accept-language":"ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7"},"remoteAddress":"0.0.0.0","userAgent":"0.0.0.0","referer":"http://127.0.0.1:5602/app/kibana"},"res":{"statusCode":500,"responseTime":9,"contentLength":9},"message":"POST /elasticsearch/_msearch 500 9ms - 9.0B"}
13

Whenever this error happened, I restart KIBANA only.
After restarting KIBANA it works for almost a day, but the error occurs again.

Anyway, please check the logs for KIBANA. I've divided the logs because of body limits.
Is there any wrong?

14

How are you running kibana? I suspect thats only a portion of the logs. Perhaps stdout and stderr are going to different logs. You can also use https://gist.github.com/ to post larger portions of content.

15

I'm running kibana by using nohup command and & for background running.

nohup kibana &

So, the logs I 've checked are from nohup.out file.
Here is the link for the logs.

https://gist.github.com/jh11425/a9bf3544faca7ba466264d11a8a79ffe

16

Honestly, still mysterious to me, but I'll keep plugging away.

I found this PR which modifies the affected behavior. Only useful for locating the error message - https://github.com/elastic/kibana/pull/28344/files/536f7d449ee32e3564bdda0be765b1b39c939b93#

How are you running Kibana and elasticsearch? Are either of them behind a proxy? Is there anything that might be modifying http headers? Can you share your kibana.yml and elasticsearch.yml content?

17

Even I am facing this issue @claire_k . I get the below error message in UI which is totally confusing :

visualise
visualise.JPG889×272 23.6 KB

More details on UI side mentioned in below url :
The aggregations key is missing from the response, check your permissions for this request

I have been searching for some inputs for somedays but could not find any valid leads.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.