Hello,
I already saw a few errors about 503 license not available. But none of these matched my error. I only see the 503 license not available randomly on a few requests. Sometimes i have to wait the 30000ms loading time, but in discover it just shows up on the right (i guess due to ajax?). When i click on try again, it works as intended.
So in general it works fine, but these waiting periods and error popups bother me. Why is this happening?
ELK Setup
- 2 master nodes in 2 (oracle virtualbox) debian vm's
- basis authentication and internode TLS communication setted up
- kibana and logstash running on on master node 1, no TLS setted up yet. Kibana uses the kibana_systems user
Screenshots
Discover:
Kibana in general:
ILM:
Kibana.log
{"type":"response","@timestamp":"2021-06-02T10:47:22+02:00","tags":[],"pid":597,"method":"post","statusCode":200,"req":{"url":"/internal/bsearch","method":"post","headers":{"host":"192.168.188.42:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://192.168.188.42:5601/app/discover","content-type":"application/json","kbn-version":"7.13.0","content-length":"1873","origin":"http://192.168.188.42:5601","dnt":"1","connection":"keep-alive","sec-gpc":"1"},"remoteAddress":"192.168.188.22","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","referer":"http://192.168.188.42:5601/app/discover"},"res":{"statusCode":200,"responseTime":442},"message":"POST /internal/bsearch 200 442ms"}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["warning","plugins","licensing"],"pid":597,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","searchprofiler"],"pid":597,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","painlessLab"],"pid":597,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","snapshotRestore"],"pid":597,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","indexManagement"],"pid":597,"message":"You cannot use index_management because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","rollup"],"pid":597,"message":"You cannot use rollup because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","indexLifecycleManagement"],"pid":597,"message":"You cannot use index_lifecycle_management because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","remoteClusters"],"pid":597,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","transform"],"pid":597,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","ingestPipelines"],"pid":597,"message":"You cannot use ingest_pipelines because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":597,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-06-02T10:47:30+02:00","tags":["warning","plugins","securitySolution"],"pid":597,"message":"Unable to verify endpoint policies in line with license change: failed to fetch package policies: missing authentication credentials for REST request [/.kibana_7.13.0/_search?from=0&rest_total_hits_as_int=true&size=100]: security_exception"}
{"type":"log","@timestamp":"2021-06-02T10:47:31+02:00","tags":["error","plugins","security","authentication"],"pid":597,"message":"License is not available, authentication is not possible."}
{"type":"log","@timestamp":"2021-06-02T10:47:36+02:00","tags":["error","plugins","security","authentication"],"pid":597,"message":"License is not available, authentication is not possible."}
{"type":"log","@timestamp":"2021-06-02T10:47:37+02:00","tags":["warning","plugins","licensing"],"pid":597,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2021-06-02T10:47:37+02:00","tags":["warning","plugins","monitoring","monitoring"],"pid":597,"message":"X-Pack Monitoring Cluster Alerts will not be available: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2021-06-02T10:48:01+02:00","tags":["warning","plugins","licensing"],"pid":597,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"response","@timestamp":"2021-06-02T10:47:31+02:00","tags":[],"pid":597,"method":"post","statusCode":503,"req":{"url":"/internal/bsearch","method":"post","headers":{"host":"192.168.188.42:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://192.168.188.42:5601/app/discover","content-type":"application/json","kbn-version":"7.13.0","content-length":"1793","origin":"http://192.168.188.42:5601","dnt":"1","connection":"keep-alive","sec-gpc":"1"},"remoteAddress":"192.168.188.22","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","referer":"http://192.168.188.42:5601/app/discover"},"res":{"statusCode":503,"responseTime":30006,"contentLength":86},"message":"POST /internal/bsearch 503 30006ms - 86.0B"}
{"type":"response","@timestamp":"2021-06-02T10:47:36+02:00","tags":[],"pid":597,"method":"post","statusCode":503,"req":{"url":"/api/ui_counters/_report","method":"post","headers":{"host":"192.168.188.42:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://192.168.188.42:5601/app/discover","content-type":"application/json","kbn-version":"7.13.0","kbn-system-request":"true","origin":"http://192.168.188.42:5601","content-length":"154","dnt":"1","connection":"keep-alive","sec-gpc":"1"},"remoteAddress":"192.168.188.22","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","referer":"http://192.168.188.42:5601/app/discover"},"res":{"statusCode":503,"responseTime":24645,"contentLength":86},"message":"POST /api/ui_counters/_report 503 24645ms - 86.0B"}
{"type":"log","@timestamp":"2021-06-02T10:48:11+02:00","tags":["error","plugins","security","authentication"],"pid":597,"message":"License is not available, authentication is not possible."}
{"type":"log","@timestamp":"2021-06-02T10:48:11+02:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":597,"message":"Starting monitoring stats collection"}
{"type":"response","@timestamp":"2021-06-02T10:48:11+02:00","tags":[],"pid":597,"method":"post","statusCode":503,"req":{"url":"/internal/bsearch","method":"post","headers":{"host":"192.168.188.42:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://192.168.188.42:5601/app/discover","content-type":"application/json","kbn-version":"7.13.0","content-length":"1793","origin":"http://192.168.188.42:5601","dnt":"1","connection":"keep-alive","sec-gpc":"1"},"remoteAddress":"192.168.188.22","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","referer":"http://192.168.188.42:5601/app/discover"},"res":{"statusCode":503,"responseTime":53,"contentLength":86},"message":"POST /internal/bsearch 503 53ms - 86.0B"}
{"type":"response","@timestamp":"2021-06-02T10:48:22+02:00","tags":[],"pid":597,"method":"post","statusCode":200,"req":{"url":"/internal/bsearch","method":"post","headers":{"host":"192.168.188.42:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://192.168.188.42:5601/app/discover","content-type":"application/json","kbn-version":"7.13.0","content-length":"1793","origin":"http://192.168.188.42:5601","dnt":"1","connection":"keep-alive","sec-gpc":"1"},"remoteAddress":"192.168.188.22","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0","referer":"http://192.168.188.42:5601/app/discover"},"res":{"statusCode":200,"responseTime":223},"message":"POST /internal/bsearch 200 223ms"}
{"type":"log","@timestamp":"2021-06-02T10:48:37+02:00","tags":["warning","plugins","licensing"],"pid":597,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2021-06-02T10:48:37+02:00","tags":["warning","plugins","monitoring","monitoring"],"pid":597,"message":"X-Pack Monitoring Cluster Alerts will not be available: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["warning","plugins","licensing"],"pid":597,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","searchprofiler"],"pid":597,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","painlessLab"],"pid":597,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","snapshotRestore"],"pid":597,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","indexManagement"],"pid":597,"message":"You cannot use index_management because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","rollup"],"pid":597,"message":"You cannot use rollup because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","indexLifecycleManagement"],"pid":597,"message":"You cannot use index_lifecycle_management because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","remoteClusters"],"pid":597,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","transform"],"pid":597,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","ingestPipelines"],"pid":597,"message":"You cannot use ingest_pipelines because license information is not available at this time."}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":597,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-06-02T10:49:00+02:00","tags":["warning","plugins","securitySolution"],"pid":597,"message":"Unable to verify endpoint policies in line with license change: failed to fetch package policies: missing authentication credentials for REST request [/.kibana_7.13.0/_search?from=0&rest_total_hits_as_int=true&size=100]: security_exception"}