Hi Team,
We have Kibana 6.8.6 (single node) connected with a coordinating only node of Elastic search (we have 3 data nodes). We are not using any proxy etc. and access Kibana directly over 5601 using self signed certificate. It has been working fine since last almost 2 years.
We recently enabled xpack security and also enabled LDAP authentication (premium).
Since then Kibana process keeps crashing. When we restart it, it works fine for few hours and then again crashes.
Following are the logs:
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Trying to authenticate via header."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Authorization header is not presented."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Trying to authenticate via state."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Request has been authenticated via state."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Request has been authenticated via state."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: clientError"}
{"type":"error","@timestamp":"2020-05-31T11:28:56Z","tags":["connection","client","error"],"pid":2241,"level":"error","error":{"message":"139785636992896:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n","name":"Error","stack":"Error: 139785636992896:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n"},"message":"139785636992896:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n"}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","legacy-service"],"pid":2241,"message":"Request will be handled by proxy POST:/api/console/proxy?path=_template&method=GET."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Trying to authenticate user request to /api/console/proxy?path=_template&method=GET."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Trying to authenticate via login attempt."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Username and password not found in payload."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Trying to authenticate via header."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Authorization header is not presented."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Trying to authenticate via state."}
{"type":"log","@timestamp":"2020-05-31T11:28:56Z","tags":["debug","security","basic"],"pid":2241,"message":"Request has been authenticated via state."}
{"type":"response","@timestamp":"2020-05-31T11:28:56Z","tags":[],"pid":2241,"method":"post","statusCode":200,"req":{"url":"/api/console/proxy?path=_aliases&method=GET","method":"post","headers":{"host":"<IP>:5601","connection":"keep-alive","content-length":"0","accept":"text/plain, */*; q=0.01","kbn-version":"6.8.6","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","origin":"https://<IP>:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<IP>:5601/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.30.28.23","userAgent":"10.30.28.23","referer":"https://<IP>:5601/app/kibana"},"res":{"statusCode":200,"responseTime":435,"contentLength":9},"message":"POST /api/console/proxy?path=_aliases&method=GET 200 435ms - 9.0B"}
{"type":"log","@timestamp":"2020-05-31T11:28:58Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:28:59Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2020-05-31T11:28:59Z","tags":["license","debug","xpack"],"pid":2241,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2020-05-31T11:29:00Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2020-05-31T11:29:00Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: clientError"}
{"type":"log","@timestamp":"2020-05-31T11:29:00Z","tags":["debug","connection","econnreset"],"pid":2241,"message":"ECONNRESET: Socket was closed by the client (probably the browser) before it could be read completely"}
{"type":"log","@timestamp":"2020-05-31T11:29:00Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:03Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:04Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2020-05-31T11:29:05Z","tags":["error","elasticsearch","data"],"pid":2241,"message":"Request error, retrying\nPOST http://<IP>:9200/_msearch?rest_total_hits_as_int=true&ignore_throttled=true => socket hang up"}
{"type":"log","@timestamp":"2020-05-31T11:29:06Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:08Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:09Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2020-05-31T11:29:10Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2020-05-31T11:29:10Z","tags":["debug","connection","econnreset"],"pid":2241,"message":"ECONNRESET: Socket was closed by the client (probably the browser) before it could be read completely"}
{"type":"log","@timestamp":"2020-05-31T11:29:10Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: clientError"}
{"type":"log","@timestamp":"2020-05-31T11:29:11Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:14Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2020-05-31T11:29:15Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:18Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:19Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2020-05-31T11:29:20Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2020-05-31T11:29:20Z","tags":["debug","connection","econnreset"],"pid":2241,"message":"ECONNRESET: Socket was closed by the client (probably the browser) before it could be read completely"}
{"type":"log","@timestamp":"2020-05-31T11:29:20Z","tags":["debug","legacy-proxy"],"pid":2241,"message":"Event is being forwarded: clientError"}
{"type":"log","@timestamp":"2020-05-31T11:29:20Z","tags":["plugin","debug"],"pid":2241,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2020-05-31T11:29:21Z","tags":["error","elasticsearch","data"],"pid":2241,"message":"Request error, retrying\nPOST http://<IP>:9200/_msearch?rest_total_hits_as_int=true&ignore_throttled=true => socket hang up"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","config"],"pid":6120,"message":"Marking config path as handled: server"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","http"],"pid":6120,"message":"registering route handler for [/core]"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","root"],"pid":6120,"message":"starting root"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","config"],"pid":6120,"message":"Marking config path as handled: logging"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","server"],"pid":6120,"message":"starting server"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","http","server"],"pid":6120,"message":"starting http server"}
{"type":"log","@timestamp":"2020-05-31T11:35:22Z","tags":["debug","http","server"],"pid":6120,"message":"http server running at https://<IP>:5601"}
In the above log following is the last line before it crashed:
{"type":"log","@timestamp":"2020-05-31T11:29:21Z","tags":["error","elasticsearch","data"],"pid":2241,"message":"Request error, retrying\nPOST http://<IP>:9200/_msearch?rest_total_hits_as_int=true&ignore_throttled=true => socket hang up"}
The co-ordinating node and Kibana are on same server hosted on RHEL.
I also tried increasing the elasticsearch.requestTimeout to 60 sec but it did not help.
Please help.
Thanks.