At random times, I am seeing the cluster fail to index new documents. From logstash I am seeing the following:
[2019-05-21T09:53:02,716][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/8/index write (api)];"})
[2019-05-21T09:53:02,716][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/8/index write (api)];"})
[2019-05-21T09:53:02,716][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/8/index write (api)];"})
[2019-05-21T09:53:02,717][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/8/index write (api)];"})
The only log entries I can see from the Elasticsearch cluster nodes is the following which occurs at the same time:
[2019-05-21T09:53:02,383][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,421][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,425][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,464][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,467][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,518][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,530][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,572][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,581][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,585][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,622][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
[2019-05-21T09:53:02,638][INFO ][o.e.c.m.MetaDataMappingService] [node-03] [winlogbeat-6.7.0-2019.05.21/zu5BnzyfSzil6Z8pE5Cydg] update_mapping [doc]
Although the FORBIDDEN error seen usually shows when the cluster is low on disk space or JVM memory, they are fine with plenty of resources available (over 450GB available on 900GB cluster)
This appears to happen at random times and no explanation as to why. Even though only some indices are set to true, the full cluster is unable to be written to by Logstash.
I end up having to resolve this by manually setting the index.blocks.write to false on the affected indices.
Cluster size:
3 nodes
1021 Shards
182 indices
Anyone seen any similar behaviour or know why this would be happening?
Cheers