Hi Guys,
While play ing with ELK version 6.5.4 with trial xpack version, i have deleted the licence and since then Kibana & elasticsearch not working. I have attached the screen shot, however i upgraded the Elasticsearch after to 6.6.2 , do i need to update Kibana and logstash too.
DELETE /_xpack_licenceHey @Rocky_RK,
Have you tried restarting Kibana? Can you post your Kibana logs so we can diagnose this further?
however i upgraded the Elasticsearch after to 6.6.2 , do i need to update Kibana
It's a good idea to keep Elasticsearch and Kibana on the same version when possible. If you're able to, I'd upgrade Kibana to 6.6.2 as well, and see if that fixes your problem
Hi @Larry_Gregory,
thanks for the revert.. yes i have restarted kibana & elastic both , i see below while i tried to get the index status .. but its now coming to one node rest two node i can see the index information coming in.. However Kibana seems at the same state.
# curl -s -XGET http://127.0.0.1:9200/_cat/indices?v
{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}
{"type":"log","@timestamp":"2019-04-25T18:25:29Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":125105,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:25:29Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":125105,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:25:29Z","tags":["plugin","debug"],"pid":125105,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-04-25T18:25:30Z","tags":["warning"],"pid":125105,"kibanaVersion":"6.5.4","nodes":[{"version":"6.6.2","http":{"publish_address":"192.168.1.1:9200"},"ip":"192.168.1.1"},{"version":"6.6.2","http":{"publish_address":"192.168.1.2:9200"},"ip":"192.168.1.2"},{"version":"6.6.2","http":{"publish_address":"10.208.20.24:9200"},"ip":"10.208.20.24"}],"message":"You're running Kibana 6.5.4 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.6.2 @ 192.168.1.1:9200 (192.168.1.1), v6.6.2 @ 192.168.1.2:9200 (192.168.1.2), v6.6.2 @ 10.208.20.24:9200 (10.208.20.24)"}
{"type":"log","@timestamp":"2019-04-25T18:25:32Z","tags":["plugin","debug"],"pid":125105,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-04-25T18:25:34Z","tags":["debug","legacy","proxy"],"pid":125105,"message":"\"getConnections\" has been called."}
{"type":"ops","@timestamp":"2019-04-25T18:25:34Z","tags":[],"pid":125105,"os":{"load":[0.76220703125,0.48193359375,0.4384765625],"mem":{"total":33536593920,"free":397778944},"uptime":6046786},"proc":{"uptime":10251.025,"mem":{"rss":377511936,"heapTotal":291008512,"heapUsed":249666088,"external":2389088},"delay":3.016345977783203},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"concurrents":{"5601":0},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 238.1MB uptime: 2:50:51 load: [0.76 0.48 0.44] delay: 3.016"}
{"type":"log","@timestamp":"2019-04-25T18:25:34Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":125105,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:25:34Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":125105,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:25:35Z","tags":["plugin","debug"],"pid":125105,"message":"Checking Elasticsearch version"}Did you try restarting or upgrading yet?
Sorry, I should have been more specific -- I really need the full kibana log file to be able to diagnose this further. Can you attach that, redacting any sensitive information?
Sure, i will attach full file, yes i have upgraded Elasticsearch and restarted both service, i'm trying to update kibana as well.. i'll send you the full kibana log
kibana.log is huge 4GB ? which look s difficult to upload.. you can have last some lines.
4gb is huge! Are there any detailed error messages coming back from Elasticsearch? Can you hit Elasticsearch directly (using curl or similar)?
Yes, i can hit the elasticsearch now with curl.. like below and i see all the index status as green now. any specific command would you like to see?
curl -s -XGET http://127.0.0.1:9200/_cat/indices?vCan you try to rename your existing kibana log file so that Kibana creates a new one? Some file systems struggle at around the 4gb file size.
Also, are there any error messages in the kibana log file that look suspect to you?
I have done that, i have re-created the file, i dont see anything suspectory ..
Ok, well now that you have a fresh log file, can you restart kibana, and then once Kibana is running, attach that log file?
I have restarted , stop/started Kibana but now kibana page doesn't open.
You should have logs for Kibana though. What is in the log file?
Below are the logs:
{"type":"log","@timestamp":"2019-04-25T18:53:52Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":146827,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:53:52Z","tags":["plugin","debug"],"pid":146827,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-04-25T18:53:53Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-04-25T18:53:53Z","tags":["spaces","debug"],"pid":146827,"message":"Verifying access to space \"default\""}
{"type":"log","@timestamp":"2019-04-25T18:53:53Z","tags":["debug","legacy-service"],"pid":146827,"message":"Request will be handled by proxy GET:/app/kibana."}
{"type":"log","@timestamp":"2019-04-25T18:53:53Z","tags":["spaces","error"],"pid":146827,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"}
{"type":"response","@timestamp":"2019-04-25T18:53:53Z","tags":[],"pid":146827,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"host":"troxnixprdc01.trox.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","cache-control":"max-age=0","dnt":"1","referer":"http://troxnixprdc01.trox.com:5601/app/kibana","upgrade-insecure-requests":"1"},"remoteAddress":"192.168.45.120","userAgent":"192.168.45.120","referer":"http://troxnixprdc01.trox.com:5601/app/kibana"},"res":{"statusCode":302,"responseTime":6,"contentLength":9},"message":"GET /app/kibana 302 6ms - 9.0B"}
{"type":"log","@timestamp":"2019-04-25T18:53:53Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-04-25T18:53:53Z","tags":["debug","legacy-service"],"pid":146827,"message":"Request will be handled by proxy GET:/."}
{"type":"response","@timestamp":"2019-04-25T18:53:53Z","tags":[],"pid":146827,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"troxnixprdc01.trox.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","cache-control":"max-age=0","dnt":"1","referer":"http://troxnixprdc01.trox.com:5601/app/kibana","upgrade-insecure-requests":"1"},"remoteAddress":"192.168.45.120","userAgent":"192.168.45.120","referer":"http://troxnixprdc01.trox.com:5601/app/kibana"},"res":{"statusCode":302,"responseTime":78,"contentLength":9},"message":"GET / 302 78ms - 9.0B"}
{"type":"log","@timestamp":"2019-04-25T18:53:45Z","tags":["plugin","debug"],"pid":146827,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":146827,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":146827,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":146827,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":146827,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-04-25T18:53:47Z","tags":["plugin","debug"],"pid":146827,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-04-25T18:53:49Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-04-25T18:53:49Z","tags":["debug","legacy-service"],"pid":146827,"message":"Request will be handled by proxy GET:/app/kibana."}
{"type":"log","@timestamp":"2019-04-25T18:53:49Z","tags":["spaces","debug"],"pid":146827,"message":"Verifying access to space \"default\""}
{"type":"log","@timestamp":"2019-04-25T18:53:49Z","tags":["spaces","error"],"pid":146827,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"}
{"type":"response","@timestamp":"2019-04-25T18:53:49Z","tags":[],"pid":146827,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"host":"troxnixprdc01.trox.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","cache-control":"max-age=0","dnt":"1","referer":"http://troxnixprdc01.trox.com:5601/app/kibana","upgrade-insecure-requests":"1"},"remoteAddress":"192.168.45.120","userAgent":"192.168.45.120","referer":"http://troxnixprdc01.trox.com:5601/app/kibana"},"res":{"statusCode":302,"responseTime":20,"contentLength":9},"message":"GET /app/kibana 302 20ms - 9.0B"}
{"type":"log","@timestamp":"2019-04-25T18:53:49Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["plugin","debug"],"pid":146827,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["debug","legacy-service"],"pid":146827,"message":"Request will be handled by proxy GET:/."}{"type":"response","@timestamp":"2019-04-25T18:53:50Z","tags":[],"pid":146827,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"troxnixprdc01.trox.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","cache-control":"max-age=0","dnt":"1","referer":"http://troxnixprdc01.trox.com:5601/app/kibana","upgrade-insecure-requests":"1"},"remoteAddress":"192.168.45.120","userAgent":"192.168.45.120","referer":"http://troxnixprdc01.trox.com:5601/app/kibana"},"res":{"statusCode":302,"responseTime":8,"contentLength":9},"message":"GET / 302 8ms - 9.0B"}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["debug","legacy-proxy"],"pid":146827,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["debug","legacy-service"],"pid":146827,"message":"Request will be handled by proxy GET:/app/kibana."}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["spaces","debug"],"pid":146827,"message":"Verifying access to space \"default\""}
{"type":"log","@timestamp":"2019-04-25T18:53:50Z","tags":["spaces","error"],"pid":146827,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"}Thanks, these are helpful!
Unable to navigate to space "default", redirecting to Space Selector. Error: Saved object [space/default] not found
This tells me that Kibana was not able to create the Default Space on startup. Since I don't have the full log file, I can't see why that happened, but if you restart Kibana, it will retry. Can you restart Kibana again and see if that resolves your problem?
Thanks @Larry_Gregory for the insight , this is what i though and already restarted the kibana but nothing changed, restarting again.. else will do reboot of the machine.
same situation still after machine reboot..
Can you run the following curl command against elasticsearch and post the result?
curl http://localhost:9200/.kibana/_search?q=type:space
# curl http://localhost:9200/.kibana/_search?q=type:space
{"took":37,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.