I upgraded elasticsearch, kibana and logstash this morning, which seems to have caused the database to become read-only, as evidenced by the following log lines;
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_elasticsearch_cluster_status_5b19d0ab-a936-4fa5-852f-05fbcd4b09e7-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_logstash_version_mismatch_aec810fd-095f-4212-9eef-e264f1386cd6-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_elasticsearch_nodes_dfd70867-8209-4dfe-baca-7de432db4392-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_xpack_license_expiration_0b1a0f65-6ff7-454b-85d4-0aed4b4a1abb-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_elasticsearch_version_mismatch_e62ef3e7-3aaf-492e-b716-9b4dd052352f-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,701][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_kibana_version_mismatch_bda4cae4-728b-4f30-a344-7ff5e7a9730a-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
I'm seeing quite a few "version mismatch" errors which I find strange - yum info tells me all components are running on 6.3.0 so i'm not sure how they could be mismatched.
[root@elastic01 ~]# yum info elasticsearch
Loaded plugins: ulninfo, versionlock
Installed Packages
Name : elasticsearch
Arch : noarch
Version : 6.3.0
Release : 1
Size : 136 M
Repo : installed
From repo : elasticsearch-6.x
Summary : Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at
: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at
: https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
URL : https://www.elastic.co/
License : Elastic License
Description : Elasticsearch subproject :distribution:packages
[root@elastic01 ~]# yum info kibana
Loaded plugins: ulninfo, versionlock
Installed Packages
Name : kibana
Arch : x86_64
Version : 6.3.0
Release : 1
Size : 482 M
Repo : installed
From repo : elasticsearch-6.x
Summary : Explore and visualize your Elasticsearch data
URL : https://www.elastic.co
License : Elastic License
Description : Explore and visualize your Elasticsearch data
[root@elastic01 ~]# yum info logstash
Loaded plugins: ulninfo, versionlock
Installed Packages
Name : logstash
Arch : noarch
Epoch : 1
Version : 6.3.0
Release : 1
Size : 238 M
Repo : installed
From repo : elasticsearch-6.x
Summary : An extensible logging pipeline
URL : http://www.elasticsearch.org/overview/logstash/
License : Elastic License
Description : An extensible logging pipeline
I followed these instructions to upgrade elasticsearch. Kibana and logstash were stopped, updated, and restarted as per their instructions.
I've had this issue before when running out of disk space, currently I have 65% of disk space free so it's not due to that. On a related note, is there any way to automatically make elasticsearch writeable after running out of disk space? Clearing space on the disk does not make a difference, I must run a command manually in order to remove the read-only flag.
Checking the health of my indices shows them as "Yellow" as follows;
[root@elastic01 ~]# curl -X GET "localhost:9200/_cat/health"
1529413219 14:00:19 elasticstack yellow 1 1 11 11 0 0 5 0 - 68.8%
The progress is going up, albeit very slowly. Between 9AM this morning and now (2PM) it has climbed only 4%. This is a single node cluster, with only around 1.5GB of data currently stored - The time this node is taking to come back up seems excessive, but i'm not sure where to begin looking for a problem with this.
I can confirm that entering the following into kibana's console resolves the issue;
PUT .kibana/_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}
This is a pain though, I would rather avoid having to run this manually any time elastic becomes read only. I'd rather avoid it becoming read only at all!
Thanks