Kibana Dev Tools not accessible post X-pack install

gingeranyhow · February 5, 2018, 1:10am

Hi!

I installed X-pack for ES and Kibana and disabled X-pack security in both the elasticsearch.yml and the kibana.yml files, and updated with the proper password in kibana.yml.

Elastic Search is up and running without issue, and I can access the Kibana interface fine. Kibana can see my indices (and thinks they're all red). However, when I try to access the Dev Tools portion of Kibana, it never loads and I get the Terminal error output below.

I'm contemplating uninstalling X-pack since it was working fine before installation. Does anyone have any other advice?

error  [00:51:44.822] [warning][kibana-monitoring][monitoring-ui] [search_phase_execution_exception] all shards failed :: {"path":"/.kibana/_search","query":{"size":0,"from":0},"body":"{\"version\":true,\"query\":{\"bool\":{\"filter\":[{\"term\":{\"type\":\"search\"}}]}}}","statusCode":503,"response":"{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}"}
at respond (/usr/local/Cellar/kibana/6.1.2/node_modules/elasticsearch/src/lib/transport.js:295:15)
at checkRespForFailure (/usr/local/Cellar/kibana/6.1.2/node_modules/elasticsearch/src/lib/transport.js:254:7)
at HttpConnector.<anonymous> (/usr/local/Cellar/kibana/6.1.2/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)
at IncomingMessage.bound (/usr/local/Cellar/kibana/6.1.2/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)
  log   [00:51:44.827] [warning][kibana-monitoring][monitoring-ui] Unable to fetch data from kibana collector
Unhandled rejection TypeError: Cannot read property 'result' of undefined
    at /usr/local/Cellar/kibana/6.1.2/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:90:50
    at Array.filter (native)
    at /usr/local/Cellar/kibana/6.1.2/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:90:29
    at next (native)
    at step (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:18:202)
    at /usr/local/Cellar/kibana/6.1.2/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:18:383
    at tryCatcher (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/util.js:11:23)
    at Promise._settlePromiseFromHandler (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:489:31)
    at Promise._settlePromise (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:546:18)
    at Promise._settlePromise0 (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
    at Promise._settlePromises (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
    at Promise._fulfill (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)
    at MappingPromiseArray.PromiseArray._resolve (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise_array.js:125:19)
    at MappingPromiseArray._promiseFulfilled (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/map.js:97:18)
    at Promise._settlePromise (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:551:26)
    at Promise._settlePromise0 (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
    at Promise._settlePromises (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
    at Promise._fulfill (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)
    at Promise._settlePromise (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:559:21)
    at Promise._settlePromise0 (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
    at Promise._settlePromises (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
    at Promise._fulfill (/usr/local/Cellar/kibana/6.1.2/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)

jpcarey · February 5, 2018, 2:49am

You are getting a 503 error when asking elasticsearch for data from the .kibana index. The elasticsearch log should have more details for the error that is causing a 503.

gingeranyhow · February 5, 2018, 5:40pm

Thanks @jpcarey - looking into it now. Getting some shard failures for sure.

[2018-02-05T09:38:52,828][WARN ][r.suppressed             ] path: /.kibana/_search, params: {size=0, index=.kibana, from=0}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:274) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:132) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:243) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:205) ~[elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:184) [elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.1.2.jar:6.1.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

gingeranyhow · February 5, 2018, 5:55pm

One note: I think it's just my kibana index - the index with all my data is still performing fine.

gingeranyhow · February 5, 2018, 6:19pm

I uninstalled X-pack and am still having this issue. Investigating that perhaps the slow log settings I changed are the issue.

gingeranyhow · February 5, 2018, 6:34pm

Hi - no need for further response. I've uninstalled and reinstalled Elastic Search and am moving forward with a clean install.

system · March 5, 2018, 6:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.