Plugin:elasticsearch@6.2.4 - Status changed from yellow to red

Rubens_Zimbres · May 22, 2018, 5:22pm

I created a pipeline in AWS where Elasticsearch sends JSON data from AWS IoT to Kibana. I configured the kibana.yml file as follows:

server.port: 5601
server.host: "localhost"
elasticsearch.url: "https://search-domain-1234-abcdef.us-east-1.es.amazonaws.com/"
elasticsearch.requestTimeout: 30000

I am able to open Kibana in my browser on http://localhost:5601 but I get the following error at Kibana page:

Request Timeout after 3000ms

Kibana_error

Also, some issues in the command line:

log [17:16:27.406] [error][status][plugin:elasticsearch@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms

respons [17:16:27.741] GET / 200 70ms - 9.0B
ops [17:16:28.590] memory: 78.7MB uptime: 0:00:06 load: [2.04 1.48 1.25] delay: 3.994
ops [17:16:33.590] memory: 79.1MB uptime: 0:00:11 load: [2.12 1.50 1.25] delay: 1.455
ops [17:16:38.589] memory: 79.4MB uptime: 0:00:16 load: [2.19 1.53 1.26] delay: 0.598
ops [17:16:43.593] memory: 79.5MB uptime: 0:00:21 load: [2.18 1.54 1.27] delay: 0.248
ops [17:16:48.596] memory: 79.7MB uptime: 0:00:26 load: [2.24 1.56 1.28] delay: 0.502
ops [17:16:53.598] memory: 79.9MB uptime: 0:00:31 load: [2.70 1.67 1.31] delay: 0.500
respons [17:16:27.850] GET /app/kibana 503 30162ms - 9.0B
respons [17:16:58.026] GET /bundles/vendors.style.css?v=16627 304 27ms - 9.0B
respons [17:16:58.031] GET /bundles/status_page.style.css?v=16627 304 27ms - 9.0B
respons [17:16:58.055] GET /ui/favicons/favicon-32x32.png 304 10ms - 9.0B
respons [17:16:58.029] GET /bundles/commons.style.css?v=16627 304 39ms - 9.0B
respons [17:16:58.069] GET /ui/favicons/favicon-16x16.png 304 3ms - 9.0B
respons [17:16:58.031] GET /bundles/vendors.bundle.js?v=16627 304 59ms - 9.0B
respons [17:16:58.140] GET /bundles/commons.bundle.js?v=16627 304 5ms - 9.0B
respons [17:16:58.162] GET /bundles/status_page.bundle.js?v=16627 304 2ms - 9.0B
respons [17:16:58.315] GET /api/status 200 4ms - 9.0B
respons [17:16:58.335] GET /plugins/kibana/assets/discover.svg 304 1ms - 9.0B
respons [17:16:58.337] GET /plugins/kibana/assets/visualize.svg 304 10ms - 9.0B
respons [17:16:58.338] GET /plugins/kibana/assets/dashboard.svg 304 12ms - 9.0B
respons [17:16:58.338] GET /plugins/timelion/icon.svg 304 20ms - 9.0B
respons [17:16:58.344] GET /plugins/kibana/assets/wrench.svg 304 16ms - 9.0B
respons [17:16:58.345] GET /plugins/kibana/assets/settings.svg 304 17ms - 9.0B
respons [17:16:58.359] GET /ui/favicons/favicon-32x32.png 304 5ms - 9.0B
respons [17:16:58.359] GET /plugins/kibana/assets/play-circle.svg 304 7ms - 9.0B
respons [17:16:58.375] GET /ui/favicons/favicon-16x16.png 304 2ms - 9.0B

I'm not using X-Pack so far.

Can someone bring some light on this issue?

Thanks in advance.

eedugon · May 22, 2018, 7:30pm

Hi @Rubens_Zimbres,

I might be wrong, but it looks to me that somehow your kibana instance is not able to connect with Elasticsearch.
Have you tried to curl from kibana terminal towards the endpoint https://search-domain-1234-abcdef.us-east-1.es.amazonaws.com/? all good? Is elasticsearch answering there?

You are not using x-pack but the endpoint is HTTPS. Is the https endpoint some kind of proxy or AWS load balancer? If elasticsearch is really listening in HTTPS then x-pack is being used, and in such case you would need x-pack configured in kibana also.

If you have direct access between kibana and elasticsearch VMs you could also try setting the endpoint to the elasticsearch service itself, not via any kind of proxy/load balancer (http://ip_elasticsearch:9200 probably).

If nothing helps feel free to share the logs from kibana since you start the service.

Regards!
Eduardo

Rubens_Zimbres · May 24, 2018, 1:17pm

Hi @eedugon, thanks for the prompt response. I successfully curled from ElasticSearch endpoint:

{
"name" : "12345y",
"cluster_name" : "1234567:domain",
"cluster_uuid" : "abcd12345",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-28T15:42:08.616107Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

However when i run bin/kibana I get:

log [13:32:53.130] [info][status][plugin:kibana@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:53.176] [info][status][plugin:elasticsearch@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.181] [info][status][plugin:xpack_main@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.188] [info][status][plugin:searchprofiler@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.196] [info][status][plugin:ml@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.264] [info][status][plugin:tilemap@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.268] [info][status][plugin:watcher@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.306] [info][status][plugin:license_management@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:53.475] [info][status][plugin:timelion@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:53.478] [info][status][plugin:graph@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:53.485] [info][status][plugin:monitoring@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:54.265] [info][status][plugin:reporting@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:54.289] [info][status][plugin:security@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [13:32:54.290] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.

log [13:32:54.331] [info][status][plugin:grokdebugger@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:54.341] [info][status][plugin:dashboard_mode@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:54.345] [info][status][plugin:logstash@6.2.4] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:32:54.363] [info][status][plugin:apm@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:54.374] [info][status][plugin:console@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:54.377] [info][status][plugin:metrics@6.2.4] Status changed from uninitialized to green - Ready
log [13:32:54.394] [info][listening] Server running at http://localhost:5601

log [13:32:56.509] [error][status][plugin:xpack_main@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.511] [error][status][plugin:searchprofiler@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.513] [error][status][plugin:ml@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.514] [error][status][plugin:tilemap@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.515] [error][status][plugin:watcher@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.516] [error][status][plugin:graph@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.518] [error][status][plugin:reporting@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.519] [error][status][plugin:security@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.521] [error][status][plugin:logstash@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:32:56.522] [error][status][plugin:elasticsearch@6.2.4] Status changed from yellow to red - Request Timeout after 3000ms
log [13:33:26.531] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms
log [13:33:26.535] [error][status][plugin:xpack_main@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.540] [error][status][plugin:searchprofiler@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.556] [error][status][plugin:ml@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.561] [error][status][plugin:tilemap@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.565] [error][status][plugin:watcher@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.573] [error][status][plugin:graph@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.582] [error][status][plugin:reporting@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.586] [error][status][plugin:security@6.2.4] Status changed from red to red - Request Timeout after 30000ms
log [13:33:26.590] [error][status][plugin:logstash@6.2.4] Status changed from red to red - Request Timeout after 30000ms

So far I just altered cluster.name in elasticsearch.yml

I set up passwords at kibana.yml:
elasticsearch.url: "https://search-domain12345.us-east-1.es.amazonaws.com/"
elasticsearch.password: secret1234
xpack.monitoring.elasticsearch.password: secret226677

And cluster.name at elasticsearch.yml

Now, Kibana at localhost:5601 tells me:

"Login is currently disabled. Administrators should consult the Kibana logs for more details."

When I run:

curl 'http://localhost:9200/?pretty'

I get:

{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

Also:
curl -u kibana:changeme http://localhost:9200/_xpack
curl: (7) Failed to connect to localhost port 9200: Connection refused

No firewall running on ports 5601 an 9200

eedugon · May 25, 2018, 10:37am

Hi!

Please try to explain the architecture a little bit more in terms of hosts and what service is running in each host.
Also share your elasticsearch.yml files from all elasticsearch nodes of the cluster.
Configuring Kibana to access Elasticsearch is quite easy, but you need to know very well your elasticsearch cluster.

Also take in mind that for production mode, SSL and X-Pack are mandatory in 6.2.x.
https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html#_development_vs_production_mode

So, as soon as you have more than 1 node in your ES cluster (or if you want to accept remote connections (from other than localhost)), you are going to need SSL certificates and use HTTPS. In case you need:

https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html

Anyway, the error reported in your output is claiming that x-pack is not installed in Elasticsearch cluster:

log [13:33:26.531] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms

I would recommend to install X-pack in your ES cluster and setup the password for the main builtin users, as explained here:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/installing-xpack-es.html

In short:

Stop Elasticsearch in all cluster nodes.
Install x-pack plugin in all your cluster nodes.

bin/elasticsearch-plugin install x-pack

Start Elasticsearch in all nodes.
Setup the basic passwords for elastic, kibana and logstash_system users.

bin/x-pack/setup-passwords interactive

Then, configure kibana with the following settings:

elasticsearch.username: "kibana"
elasticsearch.password: "your_password"
elasticsearch.url: "http://localhost:9200"

Note: This configuration is only valid if ES is a single-node cluster and running in the same host as kibana. If this is NOT the case, then you should configure SSL and certificates properly.

Regards!
Eduardo

Rubens_Zimbres · May 25, 2018, 8:53pm

@eedugon thanks for the detailed explanation. In fact I'm collecting CPU temperature from my Python notebook, sending to AWS IoT Core via MQTT using the notebook. Inside the Pyhton notebook I run Ubuntu command line 'sensors', that collects CPU temperature inside Python and sends to AWS IoT Core.

After that, IoT Core has a rule that sends the JSON generated by Python to AWS Elastic Search, using Kinesis' Firehose data stream. So far, all of these features were implemented successfully. The only missing part is the Kibana connection to AWS Elasticseach.

I know that I probably will have to set up a Lambda function to prepare data to Kibana, but I'm keeping simple so far.

In my elasticsearch.yml I only have set up the cluster name as:

cluster.name: 12345:domain

Nothing else.

I still have to find about the nodes. It's not in production yet, so I will be glad if I can successfully start Kibana without errors in a single node. Then I'll go to Lambda and clean data to visualize.

I set up passwords for x-pack but the detail is that I didn't stop Elasticsearch to install x-pack plugin.

In my kibana.yml I just set up:
elasticsearch.password: abcd12345 (probably == PASSWORD elastic)
xpack.monitoring.elasticsearch.password: abcd12345 (probably == PASSWORD logstash_system)

But I didn't insert "PASSWORD kibana".

Christian_Dahlqvist · May 26, 2018, 5:25am

Based on the URL it looks like you are using AWS Elasticsearch service, is that correct? If this is the case you can disregard settings around X-Pack, as it is not available on that service. I am not sure whether it is possible to connect an external Kibana instance to a AWS Elasticsearch service cluster or not, but I would expect your cluster to come with a Kibana instance so you do not need to host your own separately.

I would recommend having a look at our Elastic Cloud service as an alternative, which can also be had on AWS.

eedugon · May 28, 2018, 8:12am

Hi!

Yes, I fully agree with Christian. If you are using ES as a service in AWS you don't need any extra elasticsearch service running (as it won't be able to connect to the cluster at first place).

About Kibana, you might be able to connect your own kibana service (without elasticsearch instance) directly to AWS (and without x-pack in such case). Although I have never tried.

And as Christian suggests, take a look to https://www.elastic.co/cloud

Regards!
Eduardo

Rubens_Zimbres · May 29, 2018, 3:14pm

@Christian_Dahlqvist and @eedugon I solved the whole thing in a simple way. First I was trying to run Kibana from the command line to access my AWS Elasticsearch cluster.

I found out that AWS Elasticsearch policy was set up to open web access and the Kibana link at the cluster page was not working. So, I updated the ES policy to a specific IP and everything worked fine.

Details of the project can be found in my GitHub (See Project Part 2):

system · June 26, 2018, 3:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.