Kibana url getting stuck in between

Elastic Stack Kibana
1

Hi Experts,

i have configured ELK and everything started off well however, i'm facing issues with kibana now as the kibana url not working until i restart it again and this is what very frequent now.

The errors what i see now are "not sending [kibana_settings] monitoring document because [undefined] is null or invalid."} and "message":"Received Kibana Ops event data" in the /var/log/kibana.log file. I only see the suspect monitoring document.

Kibana Version: 6.8

Below is my kibana config ..

# cat /etc/kibana/kibana.yml
server.port: 5601
server.host: 0.0.0.0
server.name: prod-elk01
elasticsearch.hosts: "http://prod-elk01:9200"
logging.dest: /var/log/kibana.log
logging.verbose: true
console.enabled: true

Kibana logs..

{"type":"log","@timestamp":"2019-06-09T02:42:46Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:46Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:46Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:46Z","tags":["plugin","debug"],"pid":428695,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-06-09T02:42:47Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":428695,"message":"Polling for reindex operations"}
{"type":"log","@timestamp":"2019-06-09T02:42:48Z","tags":["plugin","debug"],"pid":428695,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-06-09T02:42:49Z","tags":["debug","stats-collection"],"pid":428695,"message":"Fetching data from kibana_stats collector"}
{"type":"log","@timestamp":"2019-06-09T02:42:49Z","tags":["debug","stats-collection"],"pid":428695,"message":"Fetching data from kibana_settings collector"}
{"type":"log","@timestamp":"2019-06-09T02:42:49Z","tags":["debug","stats-collection"],"pid":428695,"message":"not sending [kibana_settings] monitoring document because [undefined] is null or invalid."}
{"type":"log","@timestamp":"2019-06-09T02:42:49Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Uploading bulk stats payload to the local cluster"}
{"type":"log","@timestamp":"2019-06-09T02:42:49Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Uploaded bulk stats payload to the local cluster"}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","legacy-proxy"],"pid":428695,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","legacy-proxy"],"pid":428695,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","legacy-proxy"],"pid":428695,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","legacy-proxy"],"pid":428695,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","legacy-proxy"],"pid":428695,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":428695,"message":"Received Kibana Ops event data"}
{"type":"log","@timestamp":"2019-06-09T02:42:51Z","tags":["plugin","debug"],"pid":428695,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-06-09T02:42:53Z","tags":["plugin","debug"],"pid":428695,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-06-09T02:42:56Z","tags":["debug","legacy-proxy"],"pid":428695,"message":"\"getConnections\" has been called."}

Kibana Status:

# systemctl status  kibana
kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled)
   Active: active (running) since Fri 2019-06-07 15:29:00 EDT; 3 days ago
 Main PID: 428695 (node)
   CGroup: /system.slice/kibana.service
           └─428695 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Note: However, restarting kibana it works fine.

any help will be much appreciated.

2

Hello @Karn_Gusain

This error that you are seeing in the logs is when there is no default admin email set in Kibana. I am not sure though that it is the reason why Kibana stops working.

Could you elaborate on the error, is there anything logged in the Chrome dev console?

Cheers

3

Hi sebastien,

Thanks for the revert, i have the same yml working fine for the other setup for years only difference is Kibana version which is for this case is 6.8 .

For now i have deleted all the monitoring index and since then its running file , However , i also had disabled the attribute console.enabled: false.

However, there are the logs only which i have given in the post.

4

Hi @Karn_Gusain,

I just setup Kibana 6.8.0 along with Elasticsearch 6.8.0 to try and reproduce your situation. I made my kibana.yml identical to yours.

Kibana started up fine for me on the very first attempt. I visited the Kibana UI in the browser and it's working as expected too.

I see the same not sending [kibana_settings] monitoring document because [undefined] is null or invalid. message as you do. However, note that this is a debug level message, not an error or even a warning. This is not the cause of why Kibana wasn't working for you.

Would you mind temporarily commenting out the logging.verbose: true line to reduce the noise in your Kibana logs? That will help surface any warnings or errors.

For now i have deleted all the monitoring index and since then its running file , However , i also had disabled the attribute console.enabled: false .

Perhaps we could make one change at a time to try and isolate the problem here. What happens if you comment out the console.enabled: false line and restart the Kibana server?

Also, do you recall specifically which monitoring indices you deleted?

i'm facing issues with kibana now as the kibana url not working until i restart it again

When you say "not working", could you elaborate a bit more? Exactly what are the symptoms you are seeing of Kibana not working?

Thanks,

Shaunak

5

@shaunak, thanks for the detailed answer , i really appreciate that, since i've made the two changes as i said its working fine until now. However to isolate the problem, i'll start modifying the kibana.yml with one change at a time and will update you with the results.

Another point When i say "not working" it means kibana UI don't load at all and nothing appears on the UI saying that sometimes it doesn't load at all and if the session is running and reloading it again it times out, but mostly it doesn't load itself at the first place.

6

@shaunak, after the commenting out the logging.verbose: true line last time it was running till yesterday but again got into same situation and kibana stopped showing anything on the UI however service is green mean its showing as running on the Server. However i also disabled the another console.enabled: true line yesterday and now again it got into same situation.

however, log trends are only as follows in kibana.log..

ichmond-elk01.example.com:5601","referer":"http://xyz.example.com:5601/app/kibana"},"remoteAddress":"00.00.00.00","userAgent":"00.00.00.00","referer":"http://xyz.example.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"POST /api/saved_objects/_bulk_get 200 3ms - 9.0B"}
{"type":"response","@timestamp":"2019-06-14T13:14:38Z","tags":[],"pid":322283,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/_find?type=index-pattern&per_page=10000&page=1","method":"get","headers":{"host":"xyz.example.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36","accept":"*/*","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","content-type":"application/json","dnt":"1","kbn-version":"6.8.0","referer":"http://xyz.example.com:5601/app/kibana"},"remoteAddress":"00.00.00.00","userAgent":"00.00.00.00","referer":"http://xyz.example.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"GET /api/saved_objects/_find?type=index-pattern&per_page=10000&page=1 200 3ms - 9.0B"}
{"type":"response","@timestamp":"2019-06-14T13:14:39Z","tags":[],"pid":322283,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000&page=1","method":"get","headers":{"host":"xyz.example.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36","accept":"*/*","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","content-type":"application/json","dnt":"1","kbn-version":"6.8.0","referer":"http://xyz.example.com:5601/app/kibana"},"remoteAddress":"00.00.00.00","userAgent":"00.00.00.00","referer":"http://xyz.example.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"GET /api/saved_objects/_find?type=index-pattern&fields=title&per_page=10000&page=1 200 3ms - 9.0B"}
{"type":"response","@timestamp":"2019-06-14T13:14:40Z","tags":[],"pid":322283,"method":"post","statusCode":200,"req":{"url":"/api/saved_objects/_bulk_get","method":"post","headers":{"host":"xyz.example.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36","content-length":"70","accept":"*/*","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","content-type":"application/json","dnt":"1","kbn-version":"6.8.0","origin":"http://xyz.example.com:5601","referer":"http://xyz.example.com:5601/app/kibana"},"remoteAddress":"00.00.00.00","userAgent":"00.00.00.00","referer":"http://xyz.example.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"POST /api/saved_objects/_bulk_get 200 3ms - 9.0B"}
{"type":"response","@timestamp":"2019-06-14T13:14:41Z","tags":[],"pid":322283,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true","method":"post","headers":{"host":"xyz.example.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36","content-length":"806","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","content-type":"application/x-ndjson","dnt":"1","kbn-version":"6.8.0","origin":"http://xyz.example.com:5601","referer":"http://xyz.example.com:5601/app/kibana"},"remoteAddress":"00.00.00.00","userAgent":"00.00.00.00","referer":"http://xyz.example.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":52,"contentLength":9},"message":"POST /elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true 200 52ms - 9.0B"}
{"type":"response","@timestamp":"2019-06-14T13:15:04Z","tags":[],"pid":322283,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true","method":"post","headers":{"host":"xyz.example.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36","content-length":"806","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,ko;q=0.8","content-type":"application/x-ndjson","dnt":"1","kbn-version":"6.8.0","origin":"http://xyz.example.com:5601","referer":"http://xyz.example.com:5601/app/kibana"},"remoteAddress":"00.00.00.00","userAgent":"00.00.00.00","referer":"http://xyz.example.com:5601/app/kibana"},"res":{"statusCode":200,"responseTime":63,"contentLength":9},"message":"POST /elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true 200 63ms - 9.0B"}
7

Hello , @sebastien and @shaunak in if you could have time to see with the latest log i provided , however i can see now the below errors..

{"type":"log","@timestamp":"2019-06-13T07:37:08Z","tags":["error","elasticsearch","admin"],"pid":432432,"message":"Request error, retrying\nPOST http://example-elk01.xyz.com:9200/.kibana/_search?size=20&from=0&rest_total_hits_as_int=true => connect ECONNREFUSED 192.168.11.11:9200"}
{"type":"log","@timestamp":"2019-06-13T07:37:08Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:08Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:08Z","tags":["error","elasticsearch","data"],"pid":432432,"message":"Request error, retrying\nGET http://example-elk01.xyz.com:9200/_xpack => connect ECONNREFUSED 192.168.11.11:9200"}
{"type":"log","@timestamp":"2019-06-13T07:37:08Z","tags":["warning","elasticsearch","data"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:10Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:10Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:10Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:10Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:11Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:13Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:13Z","tags":["error","elasticsearch","admin"],"pid":432432,"message":"Request error, retrying\nPOST http://example-elk01.xyz.com:9200/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms => connect ECONNREFUSED 192.168.11.11:9200"}
{"type":"log","@timestamp":"2019-06-13T07:37:13Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:13Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:13Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:16Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:16Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:18Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:19Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:21Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:22Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:23Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:23Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:23Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:24Z","tags":["warning","elasticsearch","data"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:24Z","tags":["error","elasticsearch","monitoring-ui"],"pid":432432,"message":"Request error, retrying\nGET http://example-elk01.xyz.com:9200/_xpack => connect ECONNREFUSED 192.168.11.11:9200"}
{"type":"log","@timestamp":"2019-06-13T07:37:24Z","tags":["warning","elasticsearch","monitoring-ui"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:25Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:26Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:28Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
{"type":"log","@timestamp":"2019-06-13T07:37:28Z","tags":["warning","elasticsearch","admin"],"pid":432432,"message":"Unable to revive connection: http://example-elk01.xyz.com:9200/"}
8

@sebastien and @shaunak ..

However, i see the connection to elasticsearch and kibana port are well established and service also running fine, both are well connected and even health api showing good..

$ lsof -i :9200
$ lsof -i :5601

Elasticsearch health:

$ curl -XGET 'localhost:9200/?pretty'
{
  "name" : "example-elk01.cadence.com",
  "cluster_name" : "log-cohort",
  "cluster_uuid" : "VADVAYnITuChr-vqD2-pcw",
  "version" : {
    "number" : "6.8.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "65b6179",
    "build_date" : "2019-05-15T20:06:13.172855Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
9

Any help guys.. even after moving the Kibana to different host same situation it works almost for 24 hrs and again kibana UI blanks out but works when restarted again.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.