Kibana Issues with 6.4.1


(bn8595) #1

Hi I am running below version of the kibana and facing issues.
Ubuntu 16.04
elasticsearch/stable,stable,now 6.4.1 all [installed]
filebeat/stable,now 6.4.1 amd64 [installed]
kibana/stable,now 6.4.1 amd64 [installed]
logstash/stable,stable,now 1:6.4.1-1 all [installed]

cat jvm.options
-Xms1g
-Xmx1g
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
9-:-Djava.locale.providers=COMPAT
10-:-XX:UseAVX=2

LOGS
Kibana.stdout
{"type":"response","@timestamp":"2018-09-20T10:13:08Z","tags":[],"pid":9795,"method":"get","statusCode":404,"req":{"url":"/app/kibana","method":"get","headers":{"host":"localhost:5601","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8","accept-encoding":"gzip, deflate, br","accept-language":"nl-NL,nl;q=0.9,en-NL;q=0.8,en;q=0.7,en-US;q=0.6","x-forwarded-for":"172.18.4.121","x-forwarded-host":"test.example.com","x-forwarded-server":"test.example.com","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":404,"responseTime":26,"contentLength":9},"message":"GET /app/kibana 404 26ms - 9.0B"}
{"type":"log","@timestamp":"2018-09-20T10:13:16Z","tags":["info","authentication"],"pid":9795,"message":"Authentication attempt failed: [index_not_found_exception] no such index, with { resource.type="index_expression" & resource.id="_xpack" & index_uuid="na" & index="_xpack" }"}
{"type":"log","@timestamp":"2018-09-20T10:13:16Z","tags":["license","warning","xpack"],"pid":9795,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [xpack], must not start with ''., with { index_uuid="na" & index="_xpack" } :: {"path":"/_xpack","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [xpack], must not start with ''.\",\"index_uuid\":\"na\",\"index\":\"_xpack\"}],\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [xpack], must not start with ''.\",\"index_uuid\":\"na\",\"index\":\"_xpack\"},\"status\":400}"}"}
{"type":"response","@timestamp":"2018-09-20T10:13:16Z","tags":[],"pid":9795,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"localhost:5601","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8","accept-encoding":"gzip, deflate, br","accept-language":"nl-NL,nl;q=0.9,en-NL;q=0.8,en;q=0.7,en-US;q=0.6","x-forwarded-for":"172.18.4.121","x-forwarded-host":"test.example.com","x-forwarded-server":"test.example.com","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":404,"responseTime":12,"contentLength":9},"message":"GET / 404 12ms - 9.0B"}
{"type":"log","@timestamp":"2018-09-20T10:13:29Z","tags":["license","warning","xpack"],"pid":9795,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [xpack], must not start with ''., with { index_uuid="na" & index="_xpack" } :: {"path":"/_xpack","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [xpack], must not start with ''.\",\"index_uuid\":\"na\",\"index\":\"_xpack\"}],\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [xpack], must not start with ''.\",\"index_uuid\":\"na\",\"index\":\"_xpack\"},\"status\":400}"}"}

logstash logs

[2018-09-20T16:21:00,378][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}

elasticsearch logs
[2018-09-20T13:48:25,911][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CSCo8TA][write][T#4]], exiting
[2018-09-20T13:48:25,912][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CSCo8TA][write][T#3]], exiting
[2018-09-20T13:55:48,065][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-09-20T14:04:36,683][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CSCo8TA][refresh][T#2]], exiting
[2018-09-20T14:55:11,840][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-09-20T15:03:45,266][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CSCo8TA][write][T#1]], exiting
[2018-09-20T15:03:51,288][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CSCo8TA][write][T#2]], exiting
[2018-09-20T15:42:56,540][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-09-20T15:56:47,719][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CSCo8TA][management][T#1]], exiting

Ans when I try to access the kibana URL I get below error.
Ready
plugin:elasticsearch@6.4.1 [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Please help to solve.


(Lee Drengenberg) #2

When your Elasticsearch logs show fatal errors, you don't need to bother looking at your Kibana or Logstash logs. They can't work if Elasticsearch isn't working properly.

Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

It looks like Elasticsearch can't allocate memory for some reason. What's the hardware specs?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.