I installed X-pack 6.2.2 on Elasticsearch and Kibana respectively. Then I configured the built-in user accounts with the bin/x-pack/setup-passwords interactive command. Although now I can reach the Kibana interface it is really unresponsive, sluggish and I get some red error lines in the command prompt when the Kibana server is started as follows:
log [21:24:01.840] [info][status][plugin:kibana@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:02.012] [info][status][plugin:elasticsearch@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.012] [info][status][plugin:xpack_main@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.043] [info][status][plugin:searchprofiler@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.059] [info][status][plugin:ml@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.215] [info][status][plugin:tilemap@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.231] [info][status][plugin:watcher@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.340] [info][status][plugin:license_management@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:02.418] [info][status][plugin:graph@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:02.434] [info][status][plugin:monitoring@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:20.435] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
log [21:24:20.450] [info][status][plugin:reporting@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:20.606] [info][status][plugin:security@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:20.606] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml
log [21:24:20.622] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [21:24:20.778] [info][status][plugin:grokdebugger@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:20.841] [info][status][plugin:dashboard_mode@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:20.856] [info][status][plugin:logstash@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:24:20.935] [info][status][plugin:apm@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:20.981] [info][status][plugin:console@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:20.997] [info][status][plugin:metrics@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:21.856] [info][status][plugin:timelion@6.2.2] Status changed from uninitialized to green - Ready
log [21:24:21.919] [error][status][plugin:xpack_main@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.919] [error][status][plugin:searchprofiler@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.919] [error][status][plugin:ml@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms log [21:24:21.919] [error][status][plugin:tilemap@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.919] [error][status][plugin:watcher@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.935] [error][status][plugin:graph@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.935] [error][status][plugin:reporting@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.935] [error][status][plugin:security@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.935] [error][status][plugin:logstash@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.950] [error][status][plugin:elasticsearch@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [21:24:21.950] [info][listening] Server running at http://localhost:5601
log [21:24:22.075] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: trial | status: active | expiry date: 2018-05-02T14:17:21+03:00
log [21:24:22.091] [info][status][plugin:xpack_main@6.2.2] Status changed from red to green - Ready
log [21:24:22.091] [info][status][plugin:searchprofiler@6.2.2] Status changed from red to green - Ready
log [21:24:22.106] [info][status][plugin:ml@6.2.2] Status changed from red to green - Ready
...
I have the following lines in the kibana.yml file:
This seems to work, but it does not remedy the errors I get at the start. Furthermore, Logstash is no longer working and I did not install X-pack plugin for it. As far as I read on this forum and other resources modifying the output section of the .conf file defining the pipeline as follows is sufficient:
I did not edit the pipelines.yml file or the logstash.yml file. Do I have to edit them as well? As a new ELK stack user I really appreciate your help on these matters as everything was working before x-pack.
It's hard to offer much advice without some real numbers. Kibana shouldn't be sluggish, but it depends what hardware you're running on, what the CPU looks like, whether it's slow when querying data or slow all the time, etc.
If you can provide something more definitive, then we might be able to offer suggestions.
From what you've posted, it looks like you're restarting Kibana and Elasticsearch at the same time (possibly because you're restarting the box itself). Is that the case?
Kibana will indicate a red status if it cannot connect to Elasticsearch, and it's quite likely that your ES instance takes longer to start than Kibana does. So, Kibana will start, fail to connect to ES, indicate red, and then ES will start, and Kibana will go green.
If so, then it's nothing to worry about.
Logstash is no longer working
What does "no longer working" mean exactly?
I did not install X-pack plugin for it.
That's fine, you don't need to install the X-Pack plugin for Logstash unless you want Logstash monitoring (which is a good thing to have, but is not mandatory).
You can see the specs attached to this post as a screenshot (CPU stock core speed 1.9 GHz, TurboSpeed 3.2 GHz, 4 Cores). Regarding the error lines in Kibana command prompt Elasticsearch starts by itself at Windows startup (configured as a Wındows service), so it is not a full stack restart. I get the error every time I run Kibana without stopping ES.
Regarding Logstash not working I mean that data is not uploaded to ElasticSearch and there are errors in the logs files. Logstash runs as a configured Windows service as startup, for the full log please see below:
Logstash Full Log
After a certain point the log is filled with these messages:
Encountered a retryable error. Will Retry with exponential backoff {:code=>403, :url=>"http://localhost:9200/_bulk"}
@richcollier Actually without the X-pack the stack was running fine on Windows 10. @TimV I checked the documentation and there is no predefined role for logstash_writer. There are only two roles available, namely logstash_system and logstash_admin. I think it will be sufficient to use the admin role? Will it pose any problems?
You need to create that role because we don't know which indices your logstash pipeline is going to write to, so we can't ship a suitable predefined role.
Sorry I started from step 2 in that link. Thanks for pointing it out, I managed to make it work. Now the only remaining problem is the errors Kibana get during startup. Do you have a recommendation for these as well?
log [13:27:40.771] [info][status][plugin:kibana@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:40.888] [info][status][plugin:elasticsearch@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:40.903] [info][status][plugin:xpack_main@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:40.919] [info][status][plugin:searchprofiler@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:40.950] [info][status][plugin:ml@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:41.028] [info][status][plugin:tilemap@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:41.044] [info][status][plugin:watcher@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:41.107] [info][status][plugin:license_management@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:41.185] [info][status][plugin:graph@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:41.185] [info][status][plugin:monitoring@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:54.893] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
log [13:27:54.909] [info][status][plugin:reporting@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:54.987] [info][status][plugin:security@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:54.987] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml
log [13:27:54.987] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [13:27:55.065] [info][status][plugin:grokdebugger@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:55.081] [info][status][plugin:dashboard_mode@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:55.081] [info][status][plugin:logstash@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [13:27:55.143] [info][status][plugin:apm@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:55.174] [info][status][plugin:console@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:55.190] [info][status][plugin:metrics@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:55.690] [info][status][plugin:timelion@6.2.2] Status changed from uninitialized to green - Ready
log [13:27:55.753] [error][status][plugin:xpack_main@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.768] [error][status][plugin:searchprofiler@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.768] [error][status][plugin:ml@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.768] [error][status][plugin:tilemap@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.768] [error][status][plugin:watcher@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.784] [error][status][plugin:graph@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.784] [error][status][plugin:reporting@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.784] [error][status][plugin:security@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.799] [error][status][plugin:logstash@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.799] [error][status][plugin:elasticsearch@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [13:27:55.799] [info][listening] Server running at http://localhost:5601
log [13:27:55.846] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: trial | status: active | expiry date: 2018-05-02T14:17:21+03:00
log [13:27:55.883] [info][status][plugin:xpack_main@6.2.2] Status changed from red to green - Ready
log [13:27:55.888] [info][status][plugin:searchprofiler@6.2.2] Status changed from red to green - Ready
log [13:27:55.888] [info][status][plugin:ml@6.2.2] Status changed from red to green - Ready
log [13:27:55.888] [info][status][plugin:tilemap@6.2.2] Status changed from red to green - Ready
log [13:27:55.888] [info][status][plugin:watcher@6.2.2] Status changed from red to green - Ready
log [13:27:55.888] [info][status][plugin:graph@6.2.2] Status changed from red to green - Ready
log [13:27:55.904] [info][status][plugin:reporting@6.2.2] Status changed from red to green - Ready
log [13:27:55.904] [info][status][plugin:security@6.2.2] Status changed from red to green - Ready
log [13:27:55.904] [info][status][plugin:logstash@6.2.2] Status changed from red to green - Ready
log [13:27:55.935] [info][kibana-monitoring][monitoring-ui] Stopping all Kibana monitoring collectors
log [13:27:55.951] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: trial | status: active | expiry date: 2018-05-02T14:17:21+03:00
log [13:28:05.429] [info][kibana-monitoring][monitoring-ui] Starting all Kibana monitoring collectors
log [13:28:05.429] [info][status][plugin:elasticsearch@6.2.2] Status changed from red to green - Ready
Furthermore, in order to be able to reach the data that are stored in the Elasticsearch I needed to be logged into Kibana with the built-in elastic account. I alleviated the problem by defining a logstash_reader account as in the provided link. However, even then some of the indices have yellow health (it seems that the number of primary and/or replicate shards are the culprit). Here is the output of GET /_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana j2vMz0bYQVK4reGbYBOdcA 1 0 23 7 78kb 78kb
yellow open sql_tables_ykb 6tmziUk-QXWKrDdCtrg2mw 5 1 5 0 21.2kb 21.2kb
green open .watches JbuSKAaMRv6MmDz2KQZHAw 1 0 6 0 91.9kb 91.9kb
green open .watcher-history-7-2018.04.02 aBINZfTBSuSV8Mnc4Iw69g 1 0 2826 0 3.2mb 3.2mb
yellow open sql_session pYrP2lWhTzGWB-fFnMty_A 5 1 20 17 67kb 67kb
green open .monitoring-es-6-2018.04.03 4UppzqPERVi2hEBAi0bl_A 1 0 95534 1410 63mb 63mb
green open .monitoring-es-6-2018.04.02 QxYMpPQoTi2OhFGB4YDFbg 1 0 53719 1242 30.2mb 30.2mb
yellow open logstash-2015.05.18 x3paZqEfQCe-8mXpF3T0QA 5 1 4631 0 22.5mb 22.5mb
yellow open shakespeare ebe0VjMMSlGer0FuYgaOew 5 1 111396 0 21.5mb 21.5mb
green open .security-6 RuPJzrErTdqAqiT5PKSpSQ 1 0 6 0 23.4kb 23.4kb
yellow open sql_tables_training r8UcunkKR6q1eQb8iVzIGA 5 1 7 0 31.2kb 31.2kb
green open .watcher-history-7-2018.04.03 Yx2b4YRYQa6GqtbDNz8Zng 1 0 4505 0 10mb 10mb
green open .monitoring-kibana-6-2018.04.02 Wro6mUOhRliSZHQQeXbwJQ 1 0 708 0 261.8kb 261.8kb
yellow open sql_processes_training vruZLRyvRlqBIJAbgtT6Qg 5 1 2 1 12.5kb 12.5kb
green open .triggered_watches 8GtWKxjdTn2zMwFcbtbaiQ 1 0 6 102 218.5kb 218.5kb
yellow open sql_processes_ykb rWkiLp1BQWWmc-jVPIz6VA 5 1 2 0 12.6kb 12.6kb
yellow open sql_queues_training O7iXdlOlQaS-QQHK_nXcRw 5 1 4 1 30.3kb 30.3kb
yellow open logstash-2015.05.19 aWzygIVZTzCs0m4mKUXaqA 5 1 4624 0 23.4mb 23.4mb
yellow open sql_queues_ykb c99-wfuYRxWQgdx1-gxObQ 5 1 1 0 13.1kb 13.1kb
yellow open logstash-2015.05.20 NxA7gJCOSemkhdqFYUjLFQ 5 1 4750 0 22.2mb 22.2mb
green open .monitoring-kibana-6-2018.04.03 88gULRCUT2SFNMOizylfUw 1 0 937 0 616.3kb 616.3kb
yellow open logstash-2018.03.21 QETk3GO3SNeiqua2i144pQ 5 1 5 0 21.9kb 21.9kb
yellow open bank j9D6ADrMRx-pNJHpM8KVLg 5 1 1000 0 475.1kb 475.1kb
green open .monitoring-alerts-6 Zcynu28fRmiXxAnvmTD4Fw 1 0 3 0 24kb
{
"cluster_name": "elasticsearch",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 76,
"active_shards": 76,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 65,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 53.90070921985816
}
By the way Kibana continues raising timeout errors on every start (without restarting ES). I tried to configure the following lines in kibana.yml file, but they did not remedy the issue.
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
elasticsearch.requestTimeout: 30000
The command prompt still prints out timeout after 3000 miliseconds which is strange.
log [19:43:09.947] [error][status][plugin:xpack_main@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.963] [error][status][plugin:searchprofiler@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.963] [error][status][plugin:ml@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms log [19:43:09.963] [error][status][plugin:tilemap@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.963] [error][status][plugin:watcher@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.963] [error][status][plugin:graph@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.978] [error][status][plugin:reporting@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.978] [error][status][plugin:security@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.978] [error][status][plugin:logstash@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
log [19:43:09.978] [error][status][plugin:elasticsearch@6.2.2] Status changed from yellow to red - Request Timeout after 3000ms
There's a bit more to it than that.
Kibana's data security is entirely dependent on Elasticsearch's security model. If you want to view data in Kibana, then you need to have access to that data in Elasticsearch.
The only user that we ship that has out-of-the-box access to data is elastic.
This is a superuser that can do everything.
You can use that user for everything you want to do, and always login to Kibana as elastic, but we don't recommend it. Because that user can do everything, it can make a horrible mess of your cluster if you're not careful, and one of the benefits of X-Pack security is that it can protect you from mistakes like that.
Rather, we recommend that you use the elastic user to login the first time, and then use the Kibana admin screens to create new, lower privileged users and roles that have just the permissions that you need, but nothing more. You can then safely use those users to do your work in Kibana, and you can always login as elastic if you need to make major changes.
some of the indices have yellow health
An index is yellow if all primaries are available, but some (or all) replicas are missing.
It looks like you have a single node in your cluster, so it is not possible for the cluster to assign any replicas.
By the way Kibana continues raising timeout errors on every start
This is strange, but it doesn't seem to be causing any real problems, so I don't think it's anything to be worried about. You could ask in the Kibana forum if you're worried.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
