Hi All,
My Kibana Dashboard is not displaying any data after applying new license.
Please suggest some trouble shooting steps to fix this
Hi!
Can you please let me know which version of Kibana and which license you applied?
Thanks!
Hi Liza,
Thanks for the reply. I am using kibana-6.3.0-linux-x86_64. Basically idexing have been stopped due to which I am unable to see any thing in the dashboard.
Are there any error messages in the elasticsearch or kibana logs ?
Below are the error messages in log
{"type":"log","@timestamp":"2019-03-22T11:35:36Z","tags":["error","elasticsearch","admin"],"pid":7087,"message":"Request error, retrying\nPOST https://xxxx.xxx..com:9200/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms => connect ECONNREFUSED 10.68.34.192:9200"}
{"type":"log","@timestamp":"2019-03-22T11:36:01Z","tags":["license","warning","xpack"],"pid":7087,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2019-03-22T11:36:01Z","tags":["license","warning","xpack"],"pid":7087,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [monitoring] cluster. Error: No Living connections"}
{"type":"error","@timestamp":"2019-03-22T11:36:35Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":7087,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]] :: {"path":"/.kibana/doc/config%3A6.3.0","query":{},"statusCode":503,"response":"{\"error\":{\"root_cause\":[{\"type\":\"no_shard_available_action_exception\",\"reason\":\"No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]]\"}],\"type\":\"no_shard_available_action_exception\",\"reason\":\"No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]]\"},\"status\":503}"}\n at respond (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector. (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickCallback (internal/process/next_tick.js:180:9)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]]"}
{"type":"error","@timestamp":"2019-03-22T11:36:35Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":7087,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":503,"response":"{\"error\":{\"root_cause\":,\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":},\"status\":503}"}\n at respond (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector. (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickCallback (internal/process/next_tick.js:180:9)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"error","@timestamp":"2019-04-05T09:09:09Z","tags":["error","monitoring-ui"],"pid":7087,"level":"error","error":{"message":"Unable to find the cluster in the selected time range. UUID: awer0M4lS0qto8ZBlxcq7A","name":"Error","stack":"Error: Unable to find the cluster in the selected time range. UUID: awer0M4lS0qto8ZBlxcq7A\n at getClustersFromRequest (/home/unity/kibana-6.3.0-linux-x86_64/node_modules/x-pack/plugins/monitoring/server/lib/cluster/get_clusters_from_request.js:57:13)\n at "},"message":"Unable to find the cluster in the selected time range. UUID: awer0M4lS0qto8ZBlxcq7A"}
So elasticsearch and kibana were up and active before applying the new license ? If so, which license did you have before and which license are you applying?
Yes, It was running before applying the new license. I am not sure which license we were using earlier as I have not applied the new license and the one who has applied the license has left the company
Can you hit this endpoint on your elasticsearch and give me just the output of the 'status' and 'type' fields? GET /_license
Also output of just hitting your elasticsearch end point: https:://host:9200 and also GET _cluster/health.
GET _xpack/license
{
"license": {
"status": "active",
"uid": "3ad0531b-d744-4aa1-8945-9738bae2fe2c",
"type": "basic",
"issue_date": "2018-07-11T07:42:26.592Z",
"issue_date_in_millis": 1531294946592,
"max_nodes": 1000,
"issued_to": "ccb",
"issuer": "elasticsearch",
"start_date_in_millis": -1
}
}
GET _cluster/health.
{
"cluster_name": "ccb",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 16,
"active_shards": 16,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 10,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 61.53846153846154
}
curl -v telnet://sindceaplxd32:9200
- About to connect() to sindceaplxd32 port 9200 (#0)
- Trying 10.80.11.80... connected
- Connected to sindceaplxd32 (10.80.11.80) port 9200 (#0)
Thanks, can you also give me output of kibana endpoint at api/status ?
It appears to me that the license might have been downgraded to basic and some features are lost doing that, when you say your data is not displaying, is there any error displayed in the browser or console of browser? Which type of data is specifically is missing?
Can please give me the command to check output of kibana endpoint at api/status.
No I dont get any error in browser but whenever I run the query to serach logs it displays nothing.
Sure it should be something similar to http://:5601/api/status
Maybe we should also check the time and timezone, @Bargs can you help take a look see what could be wrong here?
From the logs it looks like Kibana cannot connect to the data cluster. Are you using tribe nodes? If so, have you updated the license on every node?
No I am not using any tribe node
Can you please suggest what can be done
When submitting the query in Discover, could you open the network tab of your browser's dev tools and see what response you're getting from Elasticsearch?
Could you copy what's in the "Response" tab?