Kibana Discovery Errors followed by unknown node.js issues

tomervain · November 15, 2019, 12:11pm

Hello,

I'm having some issues with Kibana recently and could really use the community's help.

I'm running Elasticsearch + Kibana 6.5.4 (both) on a Windows Server with one master node without any issues. Elasticsearch is running as a Windows Service and Kibana is running by the Kibana.bat script.

About 2 weeks ago i started to notice sudden Timeout Errors (after 30000ms) after a 12-hour (approximately) time period of proper behavior. Checking Elasticsearch's service, i didn't find any issues and Kibana was running since i could see everything from the Web App.

After checking the running Kibana.bat (via Terminal/CMD) i noticed the following non-stop errors:

Error: Cannot find module '/proc/self/environ'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at Function.Module._preloadModules (module.js:754:12)
    at preloadModules (bootstrap_node.js:488:38)
    at startup (bootstrap_node.js:201:9)
    at bootstrap_node.js:625:3
module.js:550
    throw err;
    ^

I Restarted Kibana (closed and reinitialized Kibana.bat) and everything went back to normal until it came back after the stated time period (happens every time).

I've tried to perform some basic troubleshooting:

Restarting Elasticsearch Service
Restarting the Server
Downgrade Browser's (Google Chrome) version
Replacing Kibana 6.5.4's copy with a fresh download

Nothing worked. I also tried to replace the node.js executable without success (errors seems to be generated by node's process).

I've also routed the logging to a file in order to find more information regarding thid issue and encounter this error, which i'm not sure it's related:

    {
  "type": "error",
  "@timestamp": "2019-11-14T21:10:50Z",
  "tags": [
    "warning",
    "process"
  ],
  "pid": 4668,
  "level": "error",
  "error": {
    "message": "Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6)",
    "name": "UnhandledPromiseRejectionWarning",
    "stack": "Error: Specified object should be an Object or Array
       at validateObject (KIBANA_PATH/kibana-6.5.4-windows-x86_64/kibana-6.5.4-windows-x86_64/node_modules/x-pack/server/lib/key_case_converter.js:39:11)
       at convertKeysToSnakeCaseDeep (KIBANA_PATH/kibana-6.5.4-windows-x86_64/kibana-6.5.4-windows-x86_64/node_modules/x-pack/server/lib/key_case_converter.js:44:3)
       at handler (KIBANA_PATH/kibana-6.5.4-windows-x86_64/kibana-6.5.4-windows-x86_64/node_modules/x-pack/plugins/xpack_main/server/routes/api/v1/xpack_info.js:22:13)
       at Object.internals.handler (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\handler.js:96:36)
       at request._protect.run (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\handler.js:30:23)
       at module.exports.internals.Protect.internals.Protect.run (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\protect.js:64:5)
       at exports.execute (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\handler.js:24:22)
       at each (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\request.js:384:16)
       at iterate (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:36:13)
       at done (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:28:25)
       at Hoek.once (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\protect.js:52:16)
       at wrapped (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hoek\\lib\\index.js:879:20)
       at done (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:31:25)
       at Function.wrapped [as _next] (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hoek\\lib\\index.js:879:20)
       at Function.internals.continue (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\reply.js:108:10)
       at method (KIBANA_PATH/kibana-6.5.4-windows-x86_64/kibana-6.5.4-windows-x86_64/node_modules/x-pack/plugins/dashboard_mode/server/dashboard_mode_request_interceptor.js:44:7)
       at Items.serial (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\request.js:403:22)
       at iterate (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:36:13)
       at done (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:28:25)
       at Function.wrapped [as _next] (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hoek\\lib\\index.js:879:20)
       at Function.internals.continue (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\reply.js:108:10)
       at spacesOnRequestHandler (KIBANA_PATH/kibana-6.5.4-windows-x86_64/kibana-6.5.4-windows-x86_64/node_modules/x-pack/plugins/spaces/server/lib/space_request_interceptors.js:82:16)
       at Items.serial (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\request.js:403:22)
       at iterate (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:36:13)
       at done (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\items\\lib\\index.js:28:25)
       at Function.wrapped (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hoek\\lib\\index.js:879:20)
       at Function.internals.continue (KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\node_modules\\hapi\\lib\\reply.js:108:10)
       at KIBANA_PATH\\kibana-6.5.4-windows-x86_64\\kibana-6.5.4-windows-x86_64\\src\\server\\http\\xsrf.js:33:26"
  },
  "message": "Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6)"
}

I didn't find neither similar nor related issues online, hoping someone can direct me to additional troubleshooting or a possible solution.

Thanks,

Tomer Vainstein.

Marius_Dragomir · November 18, 2019, 10:07pm

Can you any other errors in the Kibana or ES log?

tomervain · November 19, 2019, 11:19am

No, they are all the same as i posted.

When i try to access monitoring data from the API, i can see Node information. Any response to Indices status requests is Internal Server Error (500)

Marius_Dragomir · November 19, 2019, 12:46pm

What do you mean by the API? Requests to ES?

camathieu · November 19, 2019, 5:12pm

Hello,

We are experiencing the same issue since a few days.

Nov 18 15:58:30 elasticsearch5-logs-master05-am5 kibana[88005]: {"type":"error","@timestamp":"2019-11-18T15:58:30Z","tags":["warning","process"],"pid":88005,"level":"error","error":{"message":"Error: Specified object should be an Object or Array
at validateObject (/usr/local/kibana-6.3.1/node_modules/x-pack/server/lib/key_case_converter.js:39:11)
at convertKeysToSnakeCaseDeep (/usr/local/kibana-6.3.1/node_modules/x-pack/server/lib/key_case_converter.js:44:3)
at handler (/usr/local/kibana-6.3.1/node_modules/x-pack/plugins/xpack_main/server/routes/api/v1/xpack_info.js:22:13)
at Object.internals.handler (/usr/local/kibana-6.3.1/node_modules/hapi/lib/handler.js:96:36)
at request._protect.run (/usr/local/kibana-6.3.1/node_modules/hapi/lib/handler.js:30:23)
at module.exports.internals.Protect.internals.Protect.run (/usr/local/kibana-6.3.1/node_modules/hapi/lib/protect.js:64:5)
at exports.execute (/usr/local/kibana-6.3.1/node_modules/hapi/lib/handler.js:24:22)
at each (/usr/local/kibana-6.3.1/node_modules/hapi/lib/request.js:384:16)
at iterate (/usr/local/kibana-6.3.1/node_modules/items/lib/index.js:36:13)
at done (/usr/local/kibana-6.3.1/node_modules/items/lib/index.js:28:25)
at Hoek.once (/usr/local/kibana-6.3.1/node_modules/hapi/lib/protect.js:52:16)
at wrapped (/usr/local/kibana-6.3.1/node_modules/hoek/lib/index.js:879:20)
at done (/usr/local/kibana-6.3.1/node_modules/items/lib/index.js:31:25)
at Function.wrapped [as _next] (/usr/local/kibana-6.3.1/node_modules/hoek/lib/index.js:879:20)
at Function.internals.continue (/usr/local/kibana-6.3.1/node_modules/hapi/lib/reply.js:108:10)
at method (/usr/local/kibana-6.3.1/node_modules/x-pack/plugins/dashboard_mode/server/dashboard_mode_request_interceptor.js:44:7)
at Items.serial (/usr/local/kibana-6.3.1/node_modules/hapi/lib/request.js:403:22)
at iterate (/usr/local/kibana-6.3.1/node_modules/items/lib/index.js:36:13)
at done (/usr/local/kibana-6.3.1/node_modules/items/lib/index.js:28:25)
at Function.wrapped [as _next] (/usr/local/kibana-...

Kibana becomes unusable because of request timeouts on the GET /api/xpack/v1/info http route on Kibana.
A restart of the process fixes the issue for a moment then it reappears.

When the call is a success here is the output :

{
"features": {
"graph": {
"enable_app_link": false,
"message": "Graph is unavailable for the current basic license. Please upgrade your license.",
"show_app_link": false
},
"grokdebugger": {
"enable_api_route": true,
"enable_link": true
},
"index_management": {
"enable_links": true,
"is_available": true,
"show_links": true
},
"logstash": {
"enable_links": false,
"is_available": false,
"is_read_only": false,
"message": "Your basic license does not support Logstash pipeline management features. Please upgrade your license."
},
"ml": {
"enable_links": false,
"is_available": false,
"message": "Your basic license does not support Machine Learning. Please upgrade your license.",
"show_links": false
},
"reporting": {
"csv": {
"enable_links": true,
"show_links": true
},
"management": {
"enable_links": true,
"job_types": [
"csv"
],
"show_links": true
},
"printable_pdf": {
"enable_links": false,
"message": "Your basic license does not support PDF Reporting. Please upgrade your license.",
"show_links": false
}
},
"searchprofiler": {
"enable_app_link": true,
"show_app_link": true
},
"security": {
"allow_login": false,
"allow_role_document_level_security": false,
"allow_role_field_level_security": false,
"links_message": "Your Basic license does not support Security. Please upgrade your license.",
"show_links": false,
"show_login": false
},
"tilemap": {
"license": {
"active": true,
"uid": "XXXX-XXXXXXXXXXXXX-XXXXXX",
"valid": true
}
},
"watcher": {
"is_available": false,
"message": "Your basic license does not support Watcher. Please upgrade your license.",
"show_links": false
}
},
"license": {
"is_active": true,
"type": "basic"
}
}

tomervain · November 20, 2019, 8:09am

Thats correct, i meant using the REST API for monitoring requests.

From the comment above i see that others has similar issues.

camathieu · November 20, 2019, 10:03am

Also when the problem appears I noticed that those two paths behave differently :

The first one returns a 404

curl http://127.0.0.1:5601/api/x-pack/v1/info
{"statusCode":404,"error":"Not Found","message":"Not Found"}

The second one timeout :

curl http://127.0.0.1:5601/api/xpack/v1/info
^C

system · December 18, 2019, 10:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.