When I try to start Kibana on Windows 10 I get:
[fatal][root] { [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]
After that Kibana exists the batch script. Can someone please advise.
Hi,
Could you please share some surrounding errors?
Also check if you're cluster health is ok. For example if there is enough free disk space, etc.
You need to figure out which index is read-only then you may find Infos in the logs why this happened.
Regards,
Simon
Thanks for the reply.
The cluster is yellow and two the four nodes are at 95% diskspace. So the index is read only, but I don't understand why Kibana can not connect.
95% Free or used?
If it's 95% used and you can't free up space you could increase the storage thresholds, but that's not recommend as a solution, more a quick & dirty fix.
The error looks like kibana want to write into a index. Are there any details which index is read-only? The best would be if you post the logs from ES and Kibana around the Error.
Could you also share the results of the /_cat/indices/ API. Then we could see the status of your indices.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 U65-Ush3Tq2aOfAW-YZ3TQ 1 1 3 0 24.1kb 12kb
yellow open .kibana_task_manager _iSiv_8oRYKnU6z6dxcjfw 1 1 2 0 12.2kb 12.2kb
yellow open .monitoring-es-6-2020.04.21 sJv34_tBQqmoq64mhSfBvA 1 1 264683 278 113.1mb 113.1mb
yellow open .monitoring-es-6-2020.04.22 qbNZcp4IT_CWycwasFF9Tw 1 1 264617 208 112.9mb 112.9mb
yellow open .monitoring-es-6-2020.04.23 nxxIuMJkQHSDwAgodMoh3Q 1 1 264464 138 110.9mb 110.9mb
yellow open .monitoring-es-6-2020.04.24 lurGV_YGQB6Tklqaj_o6oQ 1 1 264613 136 109.3mb 109.3mb
yellow open .monitoring-es-6-2020.04.25 _cKD-TfeRw62BCn3EGO_NA 1 1 273805 210 115mb 115mb
yellow open .monitoring-es-6-2020.04.26 tCdXwSIwSxC5J_SMlBndwQ 1 1 276142 210 113.7mb 113.7mb
yellow open .monitoring-es-6-2020.04.27 7vCkpVm8R3atCfQ7W-zxVQ 1 1 275983 0 116.7mb 116.7mb
yellow open .monitoring-es-6-2020.04.28 6bJ0rFSLSdq8hChdCbObMQ 1 1 216526 144 109.4mb 109.4mb
green open votes_v1 rKYLgsBRRGyTu4Zg9gCI2g 5 1 4 0 38.8kb 19.4kb
green open websites_nl aa9cUi0qQzumeppjrG1vRg 6 1 10561999 29554 66.2gb 33.1gb
yellow open websites_v1 x5BJSWUzSIGNbnWElK2iKw 15 1 356773572 135818449 3tb 1.6tb
Nearer to 98% used
I think I know the next questions... shards 
.monitoring-es-6-2020.04.25 0 p STARTED 273805 115mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.25 0 r UNASSIGNED
.monitoring-es-6-2020.04.23 0 p STARTED 264464 110.9mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.23 0 r UNASSIGNED
.kibana_1 0 p STARTED 3 12kb 94.130.71.215 de-cluster-003
.kibana_1 0 r STARTED 3 12kb 94.130.71.232 de-cluster-004
.monitoring-es-6-2020.04.27 0 p STARTED 275983 116.7mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.27 0 r UNASSIGNED
websites_nl 4 p STARTED 1757221 5.5gb 138.201.128.230 de-cluster-002
websites_nl 4 r STARTED 1757221 5.5gb 94.130.71.215 de-cluster-003
websites_nl 1 p STARTED 1758378 5.4gb 138.201.128.230 de-cluster-002
websites_nl 1 r STARTED 1758378 5.4gb 94.130.71.215 de-cluster-003
websites_nl 5 p STARTED 1762955 5.5gb 94.130.71.215 de-cluster-003
websites_nl 5 r STARTED 1762955 5.5gb 94.130.71.232 de-cluster-004
websites_nl 3 r STARTED 1766947 5.5gb 94.130.71.215 de-cluster-003
websites_nl 3 p STARTED 1766947 5.5gb 94.130.71.232 de-cluster-004
websites_nl 2 r STARTED 1757172 5.5gb 138.201.128.230 de-cluster-002
websites_nl 2 p STARTED 1757172 5.5gb 94.130.71.215 de-cluster-003
websites_nl 0 r STARTED 1759326 5.5gb 94.130.71.215 de-cluster-003
websites_nl 0 p STARTED 1759326 5.5gb 94.130.71.232 de-cluster-004
.monitoring-es-6-2020.04.24 0 p STARTED 264613 109.3mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.24 0 r UNASSIGNED
.monitoring-es-6-2020.04.21 0 p STARTED 264683 113.1mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.21 0 r UNASSIGNED
.monitoring-es-6-2020.04.28 0 p STARTED 217254 109.7mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.28 0 r UNASSIGNED
votes_v1 4 p STARTED 2 9.3kb 94.130.71.215 de-cluster-003
votes_v1 4 r STARTED 2 9.3kb 94.130.71.232 de-cluster-004
votes_v1 1 p STARTED 1 4.8kb 138.201.128.230 de-cluster-002
votes_v1 1 r STARTED 1 4.8kb 94.130.71.215 de-cluster-003
votes_v1 3 r STARTED 1 4.7kb 94.130.71.215 de-cluster-003
votes_v1 3 p STARTED 1 4.7kb 94.130.71.232 de-cluster-004
votes_v1 2 p STARTED 0 259b 94.130.71.215 de-cluster-003
votes_v1 2 r STARTED 0 259b 94.130.71.232 de-cluster-004
votes_v1 0 p STARTED 0 259b 138.201.128.230 de-cluster-002
votes_v1 0 r STARTED 0 259b 94.130.71.215 de-cluster-003
.kibana_task_manager 0 p STARTED 2 12.2kb 94.130.71.215 de-cluster-003
.kibana_task_manager 0 r UNASSIGNED
.monitoring-es-6-2020.04.26 0 p STARTED 276142 113.7mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.26 0 r UNASSIGNED
websites_v1 11 p STARTED 23720955 120.3gb 138.201.128.230 de-cluster-002
websites_v1 11 r UNASSIGNED
websites_v1 12 r STARTED 23773792 103.8gb 78.46.77.37 de-cluster-001
websites_v1 12 p STARTED 23773792 103.8gb 94.130.71.232 de-cluster-004
websites_v1 1 p STARTED 23862729 106.4gb 78.46.77.37 de-cluster-001
websites_v1 1 r STARTED 23862729 106.4gb 138.201.128.230 de-cluster-002
websites_v1 13 p STARTED 23775109 120.5gb 94.130.71.215 de-cluster-003
websites_v1 13 r STARTED 23775109 120.5gb 94.130.71.232 de-cluster-004
websites_v1 6 r STARTED 23741978 114.9gb 78.46.77.37 de-cluster-001
websites_v1 6 p STARTED 23741978 120.6gb 94.130.71.215 de-cluster-003
websites_v1 5 r STARTED 23762269 110.9gb 138.201.128.230 de-cluster-002
websites_v1 5 p STARTED 23762269 114.3gb 94.130.71.232 de-cluster-004
websites_v1 4 p STARTED 23821925 96.7gb 78.46.77.37 de-cluster-001
websites_v1 4 r STARTED 23821925 98.9gb 94.130.71.232 de-cluster-004
websites_v1 9 r STARTED 23775836 121.3gb 138.201.128.230 de-cluster-002
websites_v1 9 p STARTED 23775836 116.9gb 94.130.71.215 de-cluster-003
websites_v1 10 r STARTED 23752029 117.2gb 138.201.128.230 de-cluster-002
websites_v1 10 p STARTED 23752029 115.2gb 94.130.71.232 de-cluster-004
websites_v1 3 r STARTED 23875988 100.9gb 78.46.77.37 de-cluster-001
websites_v1 3 p STARTED 23875988 105.6gb 94.130.71.215 de-cluster-003
websites_v1 2 p STARTED 23850670 106.2gb 78.46.77.37 de-cluster-001
websites_v1 2 r STARTED 23850670 100.7gb 94.130.71.232 de-cluster-004
websites_v1 7 p STARTED 23741142 113.1gb 138.201.128.230 de-cluster-002
websites_v1 7 r UNASSIGNED
websites_v1 14 r STARTED 23705095 120.5gb 78.46.77.37 de-cluster-001
websites_v1 14 p STARTED 23705095 120.5gb 94.130.71.232 de-cluster-004
websites_v1 8 r STARTED 23684293 115.5gb 138.201.128.230 de-cluster-002
websites_v1 8 p STARTED 23684293 108.5gb 94.130.71.215 de-cluster-003
websites_v1 0 r STARTED 23929762 99.6gb 78.46.77.37 de-cluster-001
websites_v1 0 p STARTED 23929762 106.7gb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.22 0 p STARTED 264617 112.9mb 94.130.71.215 de-cluster-003
.monitoring-es-6-2020.04.22 0 r UNASSIGNED
As I thought, there are multiple unassigned shards. So you need more disk space. You can add physical storage (best) or delete data, either deleting old data or maybe delete replicas if not needed.
Thanks for your answer, I appreciate them.
There 4 nodes, two of the nodes are above 95%, however 1 one of the nodes is at 81%. Is there a way to move the unassigned to the node at 81%?
Also is there an explanation why Kibana cant connect?
If you want you can route the shards manually. By default elastic search try to make the best routing. Maybe it's not possible because the replica would be on the same node as the primary.
Could you check your cluster settings, especially the setting cluster.routing.allocation.enable setting, may it's disabled.
Kibana couldnt start because it try to write data into an index. After reaching a storage threshold elastic search set the indices in an read only mode, so no new data could be added, that's the reason.
Hi Simon,
Thanks for the help, I will dive into this and check the setting tomorrow and let you know.
{
"persistent" : {
"xpack" : {
"monitoring" : {
"collection" : {
"enabled" : "true"
}
}
}
},
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "all"
}
}
}
}
}
looks good, so routing is enabled.
Your only option now is to increase the storage or delete old Data.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.