Elasticsearch plugin red request timeout to Elasticsearch

[Marjana]

Hello,
I have a 3 node ES cluster, 5.2.2. Kibana is running on one of the nodes, also 5.2.2.
Cluster state is green, and I can connect to it from all 3 nodes. Here is the state from node3 where kibana is running:
curl 'http://localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "logs_us",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 115,
"active_shards" : 230,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

But for some reason, kibana can't talk to it. In my UI, it tells me:
ui settings: elasticsearch plugin is red
plugin:elasticsearch@5.2.2: Request Timeout after 30000ms

Here is my kibana.yml:
server.port: 5601
server.host: "elk03.unica.net"
server.name: "elk03"
elasticsearch.url: "http://localhost:9200"
pid.file: /var/run/kibana/kibana.pid
xpack.security.enabled: false

This used to work fine, up until a few weeks ago. We had the Security pack installed and license has expired, which we have removed. I found in some forums I could add xpack.security.enabled: false to kibana.yml and elasticsearch.yml. It added it to kibana.yml, but it complained about it in elasticsearch.yml (plus xpack has been removed since). I have tried restarting kibana/ES. no avail (plugin goes from yellow to red). Nothing useful in the logs:
{"type":"log","@timestamp":"2017-04-26T20:15:51Z","tags":["status","plugin:elasticsearch@5.2.2","error"],"pid":31362,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

Anybody knows what could be wrong here?
Thanks

[tsullivan (Tim Sullivan)]

We had the Security pack installed and license has expired, which we have removed. I found in some forums I could add xpack.security.enabled: false to kibana.yml and elasticsearch.yml. It added it to kibana.yml, but it complained about it in elasticsearch.yml (plus xpack has been removed since)

I would recommend, instead of removing X-Pack from Elasticsearch to work around the expired license, register for a free basic license for X-Pack and keep that installed in Elasticsearch. You won't get errors on the settings to disable the Security features from Elasticsearch and Kibana that way, and you'll also get the single-cluster monitoring feature that comes as a "basic" feature of X-Pack.

If you're not interested in that, a few other questions come to mind:

• How did you uninstall X-Pack for Elasticsearch?
• Did you do it for all 3 nodes?
• Hopefully you did ./bin/elasticsearch-plugin uninstall x-pack.
  • Kibana has the same command you can try: ./bin/kibana-plugin uninstall x-pack
  • If you uninstall X-Pack from Kibana, you don't need the xpack.security.enabled setting

This might not be the cause of the timeouts though. It's odd that you can send a curl command to the ES node from the Kibana host. Take a pick from some of the suggestions above, and let's see if that changes anything - since as you said, this problem seemed to originate when the trial license expired.

[Marjana]

After spending some more time on this, it turned out our ES cluster has issues, with constant timeouts among the 3 nodes. It is because of that that Kibana shows ES plugin in red and it shows the same timeouts.
We upgraded all components to 5.3.0. We removed x-pack plugins. We still have problems with the cluster. High load on all 3 nodes, simple commands to list nodes or indices either hang or take a very long time to return. Logs show constant "o.e.t.TransportService" timeouts. But that problem is for another thread. I believe once ES is stable, Kibana plugins will work just fine. Thanks!