"license","warning","xpack"

Hi Team

I installed Elasticsearch 6.8 and kibana 6.8, with basic license, for some time both were working find. now i am unable to get my Kibana working, though the ES is green and i can see all my app logs being pumped to ES with auth enabled...

My kibana log says:
{"type":"log","@timestamp":"2019-07-29T03:41:39Z","tags":["license","warning","xpack"],"pid":28265,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2019-07-29T03:42:09Z","tags":["warning","task_manager"],"pid":28265,"message":"PollError Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2019-07-29T03:42:39Z","tags":["license","warning","xpack"],"pid":28265,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2019-07-29T03:42:42Z","tags":["warning","task_manager"],"pid":28265,"message":"PollError Request Timeout after 30000ms"}

My kibana yml:

elasticsearch.url: "http://x.x.x.x:9200"
server.host: "x.x.x.x"

elasticsearch.username: "kibana"
elasticsearch.password: "xxxxxx"

xpack.security.enabled: true
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.watcher.enabled: false
xpack.ccr.enabled: false

My elasticsearch yml:

cluster.name: "xxxxxxxxxxxxxxxxxxxxx"
node.name: ${HOSTNAME}
node.master: true
node.data: false
cluster.routing.allocation.awareness.force.zone.values: 1A, 1B
cluster.routing.allocation.awareness.attributes: zone
node.attr.zone: 1A
discovery.zen.ping.unicast.hosts: ["xxxxx", "xxxxxx", "xxxxx"]
path.data: /esdata/data/
path.logs: /esdata/logs/
network.host: x.x.x.x
discovery.zen.minimum_master_nodes: 2
http.port: 9200
gateway.recover_after_nodes: 1
bootstrap.system_call_filter: false

xpack.security.enabled: true
xpack.security.authc.accept_default_password: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

My License details:
{
"license" : {
"status" : "active",
"uid" : "d7d4d0ce-9d8d-402d-92c4-2093691bf643",
"type" : "basic",
"issue_date" : "2019-05-28T05:28:48.078Z",
"issue_date_in_millis" : 1559021328078,
"max_nodes" : 1000,
"issued_to" : "IMIConnect-Prod-Imonitor-Cluster",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}

My x-pack status:

{
"build" : {
"hash" : "65b6179",
"date" : "2019-05-15T20:07:59.571448Z"
},
"license" : {
"uid" : "d7d4d0ce-9d8d-402d-92c4-2093691bf643",
"type" : "basic",
"mode" : "basic",
"status" : "active"
},
"features" : {
"ccr" : {
"description" : "Cross Cluster Replication",
"available" : false,
"enabled" : true
},
"graph" : {
"description" : "Graph Data Exploration for the Elastic Stack",
"available" : false,
"enabled" : true
},
"ilm" : {
"description" : "Index lifecycle management for the Elastic Stack",
"available" : true,
"enabled" : true
},
"logstash" : {
"description" : "Logstash management component for X-Pack",
"available" : false,
"enabled" : true
},
"ml" : {
"description" : "Machine Learning for the Elastic Stack",
"available" : false,
"enabled" : true,
"native_code_info" : {
"version" : "6.8.0",
"build_hash" : "e6cf25e2acc5ec"
}
},
"monitoring" : {
"description" : "Monitoring for the Elastic Stack",
"available" : true,
"enabled" : true
},
"rollup" : {
"description" : "Time series pre-aggregation and rollup",
"available" : true,
"enabled" : true
},
"security" : {
"description" : "Security for the Elastic Stack",
"available" : true,
"enabled" : true
},
"sql" : {
"description" : "SQL access to Elasticsearch",
"available" : true,
"enabled" : true
},
"watcher" : {
"description" : "Alerting, Notification and Automation for the Elastic Stack",
"available" : false,
"enabled" : true
}
},
"tagline" : "You know, for X"

Can some help me on this please

I have moved this to the Kibana topic.

Your problem does not appear to have anything to do with security or licensing. Your Kibana server cannot connect to Elasticsearch.

Thanks Tim,

My cluster is hosted in AWS Cloud, its a three node cluster with two nodes as data & master, while the other node is dedicated master.

i have installed the kibana server in my master only node.,
I do see that the kibana is running when i checked the status from with in the server..

but when when i call the kibana server via browser,
it says kibana server is not ready..

"Kibana server is not ready yet"

but my cluster is green and indices are good and i can do a curl from the master node to other nodes..

From what you've written above, this is expected behaviour.

Kibana cannot connect to Elasticsearch, so it is not possible to access Kibana.

What we need to diagnose is why you are getting timeouts from Kibana to Elasticsearch when curl is working.

I would start by double checking that you can run curl from the kibana user, using exactly the same settings as in your kibana.yml (same url, same username, same password).

Hi Tim

I Tried, and its working fine..

[root@ip-172-16-190-20 ~]# curl -XGET "http://kibana:xxxxxx@x.x.x.x:9200/_cluster/health"?pretty
{
"cluster_name" : "xxxxxxxxxxxxxxxxxxxxxxxxxx",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 2,
"active_primary_shards" : 299,
"active_shards" : 594,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0

We are using a load balancer over the data nodes..,
but when i try to connect curl through load balancer..

[root@ip-172-16-190-20 ~]# curl -XGET "http://kibana:xxxxxxx
@imonitor-datanode.imiconnect.net:9200/_cluster/health"?pretty
curl: (7) Failed to connect to imonitor-datanode.imiconnect.net port 9200: Connection timed out

Now i removed calling via load balancer and used the data node ip as URL..

when i restarted kibana...i get this error:

{"type":"log","@timestamp":"2019-07-29T06:07:53Z","tags":["error","task_manager"],"pid":10914,"message":"Failed to poll for work: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana_task_manager/_doc/oss_telemetry-vis_telemetry/_update","query":{"if_seq_no":84,"if_primary_term":6,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\",\"task\":{\"taskType\":\"vis_telemetry\",\"state\":\"{\\\"runs\\\":9,\\\"stats\\\":{\\\"markdown\\\":{\\\"total\\\":1,\\\"spaces_min\\\":1,\\\"spaces_max\\\":1,\\\"spaces_avg\\\":1},\\\"table\\\":{\\\"total\\\":10,\\\"spaces_min\\\":10,\\\"spaces_max\\\":10,\\\"spaces_avg\\\":10},\\\"metric\\\":{\\\"total\\\":7,\\\"spaces_min\\\":7,\\\"spaces_max\\\":7,\\\"spaces_avg\\\":7},\\\"histogram\\\":{\\\"total\\\":3,\\\"spaces_min\\\":3,\\\"spaces_max\\\":3,\\\"spaces_avg\\\":3}}}\",\"params\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2019-07-16T03:14:57.700Z\",\"runAt\":\"2019-07-29T06:12:53.760Z\",\"status\":\"running\"},\"kibana\":{\"uuid\":\"e7d14ee1-da18-4a35-8603-e088a5e7f45c\",\"version\":6080099,\"apiVersion\":1}}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}
{"type":"log","@timestamp":"2019-07-29T06:07:55Z","tags":["security","error"],"pid":10914,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];, with { suppressed={ 0={ type="cluster_block_exception" & reason="blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];" } & 1={ type="cluster_block_exception" & reason="blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];" } & 2={ type="cluster_block_exception" & reason="blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];" } } }"}
{"type":"log","@timestamp":"2019-07-29T06:07:56Z","tags":["error","task_manager"],"pid":10914,"message":"Failed to poll for work: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana_task_manager/_doc/oss_telemetry-vis_telemetry/_update","query":{"if_seq_no":84,"if_primary_term":6,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\",\"task\":{\"taskType\":\"vis_telemetry\",\"state\":\"{\\\"runs\\\":9,\\\"stats\\\":{\\\"markdown\\\":{\\\"total\\\":1,\\\"spaces_min\\\":1,\\\"spaces_max\\\":1,\\\"spaces_avg\\\":1},\\\"table\\\":{\\\"total\\\":10,\\\"spaces_min\\\":10,\\\"spaces_max\\\":10,\\\"spaces_avg\\\":10},\\\"metric\\\":{\\\"total\\\":7,\\\"spaces_min\\\":7,\\\"spaces_max\\\":7,\\\"spaces_avg\\\":7},\\\"histogram\\\":{\\\"total\\\":3,\\\"spaces_min\\\":3,\\\"spaces_max\\\":3,\\\"spaces_avg\\\":3}}}\",\"params\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2019-07-16T03:14:57.700Z\",\"runAt\":\"2019-07-29T06:12:56.801Z\",\"status\":\"running\"},\"kibana\":{\"uuid\":\"e7d14ee1-da18-4a35-8603-e088a5e7f45c\",\"version\":6080099,\"apiVersion\":1}}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}

My node allocation is
[root@ip-172-16-190-20 ~]# curl -XGET "http://kibana:xxxxxxxxx@172.16.190.20:9200/_cat/allocation"?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
298 146.8gb 164.9gb 334.8gb 499.7gb 32 172.16.190.89 172.16.190.89 ip-172-16-190-89.eu-west-1.compute.internal
298 146.5gb 164.9gb 334.7gb 499.7gb 33 172.16.190.23 172.16.190.23 ip-172-16-190-23.eu-west-1.compute.internal

Are you running out of disk space for Elasticsearch?

Hi Christian

We ran out of disk but than i cleaned up data via curl and as given in the earlier reply i do see that my disks are only 35% used.

Then you need to remove the index block. Search for watermark in the docs for an example.

Thanks Christian and Tim

There seemed to be an issue with the aws load balancer, when we changed the Elasticsearch url to data node IP and set the curl -XPUT 'http://kibana:xxxxxxxx@x.x.x.x:9200/_settings {"index": { "blocks": { "read_only_allow_delete": "false" }}}'

then restarted Kibana., this solved the problem

Thanks for the wounderful guidance from Tim and Christian