Elasticsearch read-only after upgrade (6.2.0 -> 6.3.0)


(Jack Arnold) #1

I upgraded elasticsearch, kibana and logstash this morning, which seems to have caused the database to become read-only, as evidenced by the following log lines;

[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_elasticsearch_cluster_status_5b19d0ab-a936-4fa5-852f-05fbcd4b09e7-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_logstash_version_mismatch_aec810fd-095f-4212-9eef-e264f1386cd6-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_elasticsearch_nodes_dfd70867-8209-4dfe-baca-7de432db4392-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_xpack_license_expiration_0b1a0f65-6ff7-454b-85d4-0aed4b4a1abb-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,700][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_elasticsearch_version_mismatch_e62ef3e7-3aaf-492e-b716-9b4dd052352f-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[2018-06-19T13:51:37,701][ERROR][o.e.x.w.e.ExecutionService] [elastic01] could not store triggered watch with id [ansAPnhIS3atR9t5kZBBkA_kibana_version_mismatch_bda4cae4-728b-4f30-a344-7ff5e7a9730a-2018-06-19T12:51:37.698Z]: [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]

I'm seeing quite a few "version mismatch" errors which I find strange - yum info tells me all components are running on 6.3.0 so i'm not sure how they could be mismatched.

[root@elastic01 ~]# yum info elasticsearch
Loaded plugins: ulninfo, versionlock
Installed Packages
Name        : elasticsearch
Arch        : noarch
Version     : 6.3.0
Release     : 1
Size        : 136 M
Repo        : installed
From repo   : elasticsearch-6.x
Summary     : Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at
            : https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at
            : https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
URL         : https://www.elastic.co/
License     : Elastic License
Description : Elasticsearch subproject :distribution:packages

[root@elastic01 ~]# yum info kibana
Loaded plugins: ulninfo, versionlock
Installed Packages
Name        : kibana
Arch        : x86_64
Version     : 6.3.0
Release     : 1
Size        : 482 M
Repo        : installed
From repo   : elasticsearch-6.x
Summary     : Explore and visualize your Elasticsearch data
URL         : https://www.elastic.co
License     : Elastic License
Description : Explore and visualize your Elasticsearch data

[root@elastic01 ~]# yum info logstash
Loaded plugins: ulninfo, versionlock
Installed Packages
Name        : logstash
Arch        : noarch
Epoch       : 1
Version     : 6.3.0
Release     : 1
Size        : 238 M
Repo        : installed
From repo   : elasticsearch-6.x
Summary     : An extensible logging pipeline
URL         : http://www.elasticsearch.org/overview/logstash/
License     : Elastic License
Description : An extensible logging pipeline

I followed these instructions to upgrade elasticsearch. Kibana and logstash were stopped, updated, and restarted as per their instructions.

I've had this issue before when running out of disk space, currently I have 65% of disk space free so it's not due to that. On a related note, is there any way to automatically make elasticsearch writeable after running out of disk space? Clearing space on the disk does not make a difference, I must run a command manually in order to remove the read-only flag.

Checking the health of my indices shows them as "Yellow" as follows;

[root@elastic01 ~]# curl -X GET "localhost:9200/_cat/health"
1529413219 14:00:19 elasticstack yellow 1 1 11 11 0 0 5 0 - 68.8%

The progress is going up, albeit very slowly. Between 9AM this morning and now (2PM) it has climbed only 4%. This is a single node cluster, with only around 1.5GB of data currently stored - The time this node is taking to come back up seems excessive, but i'm not sure where to begin looking for a problem with this.

I can confirm that entering the following into kibana's console resolves the issue;

PUT .kibana/_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}

This is a pain though, I would rather avoid having to run this manually any time elastic becomes read only. I'd rather avoid it becoming read only at all!

Thanks


#2

Hey there,

I can confirm that issue with my ES 6.3.0 Test environment. For me, the problems began when installing X-Pack and trying to integrate Logstash Pipeline Management (without having the necessary license, as I found out later). But that can also be a coincidence.

Workaround is the same as you posted. There's also enough disk space left as in your case. Would be nice, if someone could take a closer look into this.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.