I am running Elk stack 7 and I am not seeing any data on the monitoring tab. Any idea how I can trouble shoot this issue.
Hi @syedsfayaz - So changing your time filter still returns no results?
If you use the dev tools and run GET _cat/indices, do you see a .monitoring-* index listed there?
I ran this command
GET _cat/indices/.monitoring-es*
green open .monitoring-es-7-2019.05.06 _aIdN2c0T1qAL1R57FqGTA 1 0 1133260 167862 319.3mb 319.3mb
green open .monitoring-es-7-2019.05.05 _KniuoHyQP63iMn8JczVTg 1 0 1133814 132191 319.4mb 319.4mb
green open .monitoring-es-7-2019.05.04 XeYC4p1LR1auZOV8MYkkIw 1 0 1134030 239190 331.5mb 331.5mb
green open .monitoring-es-7-2019.05.07 uROh4fT4Q9GXEw-9Mota1g 1 0 763842 277554 301.3mb 301.3mb
green open .monitoring-es-7-2019.05.03 AIO0eaIqRtaxa4odnu7TIA 1 0 1132667 554222 589.5mb 589.5mb
green open .monitoring-es-7-2019.05.01 89tMUXq5R9-RqPmUl991mw 1 0 1148325 99048 317.8mb 317.8mb
green open .monitoring-es-7-2019.05.02 PWUxfOLTQc2r0a4JlNXiow 1 0 1171301 408602 382.8mb 382.8mbOkay, so there's definitely monitoring data then -- do you have a .monitoring-kibana-* index as well?
Do you see any data if you adjust the time filter?
I also see data for kibana. But even after adjusting the time filter I don't see any data on dashboard.
GET _cat/indices/.monitoring-kib*
green open .monitoring-kibana-7-2019.05.03 UTEICnDbSKGF2zk4itBSaw 1 0 8639 0 2.9mb 2.9mb
green open .monitoring-kibana-7-2019.05.01 2XDsyaH2Ra2RFCfuYcOTfQ 1 0 8640 0 2.9mb 2.9mb
green open .monitoring-kibana-7-2019.05.05 341ygbesQF6XI1MnHy6tew 1 0 8639 0 2.9mb 2.9mb
green open .monitoring-kibana-7-2019.05.06 V2WBbkASR6KUsVvoRiuD2w 1 0 8639 0 2.9mb 2.9mb
green open .monitoring-kibana-7-2019.05.07 85_rKneXRFmA-DQ6eXKDDg 1 0 5884 0 2mb 2mb
green open .monitoring-kibana-7-2019.05.04 LZi3GEZAQESkZEEdLToC4g 1 0 8639 0 2.9mb 2.9mb
green open .monitoring-kibana-7-2019.05.02 Gv42QAVpS6acaViyXl4HLw 1 0 8639 0 2.9mb 2.9mb
In your kibana.yml, do you have any xpack.monitoring settings set?
I don't see any settings for xpack.monitoring. Do I need a license to enable xpack.monitoring. I was able to see this in ELK 6.7 with out adding any settings. I remember seeing the monitoring after upgrading to 7 as well. But recently its not working.
You do need a license for Monitoring. I'm pretty sure you have it, though, but just to double check can you run the following query in Kibana Console?
GET _license
I remember seeing the monitoring after upgrading to 7 as well. But recently its not working.
Interesting. Can you think of anything that changed recently, after upgrading to 7?
Finally, would you mind posting the results of this query as well?
POST .monitoring-es-*/_search
{
"query": {
"bool": {
"filter": {
"term": {
"type": "cluster_stats"
}
}
}
},
"collapse": {
"field": "cluster_uuid"
},
"sort": [
{
"timestamp": {
"order": "desc"
}
}
]
}
Thanks,
Shaunak
I upgraded my ELK stack from 6.7 to 7. I was running my beats still in 6.7 . I have upgraded few of them which are running both 7.0 and 7.1 versions. Do I need to add xpack monitoring settings in kibana.yml?
{
"license" : {
"status" : "active",
"uid" : "*********************",
"type" : "basic",
"issue_date" : "2019-03-08T08:50:34.454Z",
"issue_date_in_millis" : 1552035034454,
"max_nodes" : 1000,
"issued_to" : "elasticsearch",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
{
"took" : 15,
"timed_out" : false,
"_shards" : {
"total" : 7,
"successful" : 7,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [
{
"_index" : ".monitoring-es-7-2019.05.07",
"_type" : "_doc",
"_id" : "k2oB5GIIEeSL***",
"_score" : null,
"fields" : {
"cluster_uuid" : [
"***-Le_3uRl3"
]
},
"sort" : [
1557249986408
]
}
]
}
}No, you don't need to do this, unless you're using a separate cluster for storing your monitoring data (which it sounds like you aren't). I asked about this earlier just to understand your setup.
Looking at the results of your queries, all looks good to me. @chrisronline Any ideas why the Monitoring UI might still be showing the "no data" page in this case?
@chrisronline Can you please help me.
Are you seeing any error in the kibana log file?
I am not seeing any erros in the kibana logs.
{"type":"response","@timestamp":"2019-05-09T17:36:03Z","tags":[],"pid":11179,"method":"get","statusCode":304,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Light-BETA.woff2","method":"get","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","origin":"http://hostname.changed.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","accept":"*/*","referer":"http://hostname.changed.com:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"15e146985711bc9631d87ffaa1df8bcad2036ef3\"","if-modified-since":"Fri, 05 Apr 2019 23:21:39 GMT"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102","referer":"http://hostname.changed.com:5601/app/monitoring"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Light-BETA.woff2 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-05-09T17:36:03Z","tags":[],"pid":11179,"method":"get","statusCode":200,"req":{"url":"/api/monitoring/v1/elasticsearch_settings/check/nodes","method":"get","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","accept":"application/json, text/plain, */*","kbn-version":"7.0.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","referer":"http://hostname.changed.com:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102","referer":"http://hostname.changed.com:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /api/monitoring/v1/elasticsearch_settings/check/nodes 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2019-05-09T17:36:13Z","tags":[],"pid":11179,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","content-length":"81","accept":"application/json, text/plain, */*","origin":"http://hostname.changed.com:5601","kbn-version":"7.0.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://hostname.changed.com:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102","referer":"http://hostname.changed.com:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":24,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 200 24ms - 9.0B"}Using the cluster uuid from above, try and go to this page:
<kibana_url>/app/monitoring#/overview?_g=(cluster_uuid:<cluster_uuid>)
Check the kibana logs assuming the page doesn't work and let me know if it reports any error
I see this warning in the logs.
{"type":"log","@timestamp":"2019-05-09T17:35:42Z","tags":["reporting","browser-driver","warning"],"pid":11179,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2019-05-09T17:35:42Z","tags":["reporting","warning"],"pid":11179,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}I tried to use one of the uuid of ES in the url .
http://hostname.changed.com:5601//app/monitoring#/overview?_g=(cluster_uuid:nXGsRyZJQK2SqGf6iUSZQg)
{"statusCode":404,"error":"Not Found","message":"Not Found"}
{"type":"response","@timestamp":"2019-05-09T17:57:27Z","tags":[],"pid":11179,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_mapping&method=GET","method":"post","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","content-length":"0","accept":"text/plain, */*; q=0.01","origin":"http://hostname.changed.com:5601","kbn-version":"7.0.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","referer":"http://hostname.changed.com:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102","referer":"http://hostname.changed.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":684,"contentLength":9},"message":"POST /api/console/proxy?path=_mapping&method=GET 200 684ms - 9.0B"}
{"type":"response","@timestamp":"2019-05-09T17:58:20Z","tags":[],"pid":11179,"method":"get","statusCode":404,"req":{"url":"//app/monitoring","method":"get","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102"},"res":{"statusCode":404,"responseTime":46,"contentLength":9},"message":"GET //app/monitoring 404 46ms - 9.0B"}Try removing the double / and lemme know what you see
{"type":"response","@timestamp":"2019-05-09T18:06:43Z","tags":[],"pid":11179,"method":"get","statusCode":200,"req":{"url":"/api/spaces/space","method":"get","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","accept":"application/json, text/plain, */*","kbn-version":"7.0.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","referer":"http://hostname.changed.com:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102","referer":"http://hostname.changed.com:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"GET /api/spaces/space 200 3ms - 9.0B"}
{"type":"error","@timestamp":"2019-05-09T18:06:43Z","tags":["error","monitoring-ui"],"pid":11179,"level":"error","error":{"message":"Unable to find the cluster in the selected time range. UUID: nXGsRyZJQK2SqGf6iUSZQg","name":"Error","stack":"Error: Unable to find the cluster in the selected time range. UUID: nXGsRyZJQK2SqGf6iUSZQg\n at getClustersFromRequest (/data/elk/kibana-7/node_modules/x-pack/plugins/monitoring/server/lib/cluster/get_clusters_from_request.js:66:13)"},"message":"Unable to find the cluster in the selected time range. UUID: nXGsRyZJQK2SqGf6iUSZQg"}
{"type":"response","@timestamp":"2019-05-09T18:06:43Z","tags":[],"pid":11179,"method":"get","statusCode":304,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Medium.woff2","method":"get","headers":{"host":"hostname.changed.com:5601","connection":"keep-alive","origin":"http://hostname.changed.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36","accept":"*/*","referer":"http://hostname.changed.com:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"c4420439cbbcbdf76d78fd6ec69b4c8eb7afb2b2\"","if-modified-since":"Fri, 05 Apr 2019 23:21:39 GMT"},"remoteAddress":"10.14.51.102","userAgent":"10.14.51.102","referer":"http://hostname.changed.com:5601/app/monitoring"},"res":{"statusCode":304,"responseTime":1,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Medium.woff2 304 1ms - 9.0B"}Can you provide the full results of running this query (again):
POST .monitoring-es-*/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"type": "cluster_stats"
}
}
]
}
},
"collapse": {
"field": "cluster_uuid"
},
"sort": {
"timestamp": {
"order": "desc"
}
}
}
I'd like to see what's in the source document too. Feel free to scrub any sensitive data
{
"took" : 608,
"timed_out" : false,
"_shards" : {
"total" : 8,
"successful" : 8,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [
{
"_index" : ".monitoring-es-7-2019.05.09",
"_type" : "_doc",
"_id" : "o8bTnWoB3maiDp4b9bDG",
"_score" : null,
"fields" : {
"cluster_uuid" : [
"T3FxB08PRy-Le_3uRl3Zag"
]
},
"sort" : [
1557426075022
]
}
]
}
}