Config: Error 403 Forbidden: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Elastic Stack Kibana
1

Config: Error 403 Forbidden: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Kibana not able to retrieve old data.

I have enabled one of the device it started logging huge data, after some time Elastic Search and Kibana not showing data, i have restarted, but no luck, i see below errors

syslog-ng ---elasticksearch--kibana is my setup

[2017-12-27T20:07:44.841332] Outgoing message; message='Dec 27 20:07:44 ES6 kibana[7035]: {"type":"error","@timestamp":"2017-12-27T20:07:44Z","tags":["error","monitoring-ui"],"pid":7035,"level":"error","error":{"message":"[export_exception] Exception when closing export bulk","name":"Error","stack":"[export_exception] Exception when closing export bulk :: {"path":"/_xpack/monitoring/_bulk","query":{"system_id":"kibana","system_api_version":"6","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_settings\"}}\n{\"kibana_uuid\":\"65539add-f049-4084-800e-62ebe31172c5\",\"xpack\":{\"default_admin_email\":null}}\n{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"65539add-f049-4084-800e-62ebe31172c5\",\"name\":\"ES6\",\"index\":\".kibana\",\"host\":\"ES6\",\"transport_address\":\"192.168.1.75:5601\",\"version\":\"6.0.1\",\"snapshot\":false,\"status\":\"green\"},\"concurrent_connections\":393,\"os\":{\"load\":{\"1m\":0.03564453125,\"5m\":0.23828125,\"15m\":0.458984375},\"memory\":{\"total_in_bytes\":8371228672,\"free_in_bytes\":1713790976,\"used_in_bytes\":6657437696},\"uptime_in_millis\":1458957000},\"process\":{\"event_loop_delay\":97.87312602996826,\"memory\":{\"heap\":{\"total_in_bytes\":120438784,\"used_in_bytes\":102224128,\"size_limit\":1501560832},\"resident_set_size_in_bytes\":161980416},\"uptime_in_millis\":482334},\"requests\":{\"disconnects\":0,\"total\":205,\"status_codes\":{\"200\":109,\"304\":77,\"403\":14,\"404\":5}},\"response_times\":{\"average\":95,\"max\":699},\"timestamp\":\"2017-12-27T20:07:39.449Z\"}\n","statusCode":500,"response":"{\"took\":38,\"errors\":true,\"error\":{\"type\":\"export_exception\",\"reason\":\"Exception when closing export bulk\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulks\",\"caused_by\\x0a'
[2017-12-27T20:07:44.841386] Outgoing message; message='Dec 27 20:07:44 ES6 kibana[7035]: ":{\"type\":\"export_exception\",\"reason\":\"bulk [default_local] reports failures when exporting documents\",\"exceptions\":[{\"type\":\"export_exception\",\"reason\":\"ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused_by\":{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}},{\"type\":\"export_exception\",\"reason\":\"ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused_by\":{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}}]}}}}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:91:20)\n at IncomingMessage.emit (events.js:185:7)\n at endReadableNT (_stream_readable.js:974:12)\n at _combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickDomainCallback (internal/process/next_tick.js:128:9)"},"message":"[export_exception] Exception when closing export bulk"}\x0a'

1 Like
2

Can you verify that there's disk space available? ES will go into read only mode once a threshold is hit.

2 Likes
3

Yes i have 2+GB available free, when i see that first i checked space.

4

/dev/mapper/ES6--vg-root 12G 5.5G 5.3G 52% /

5

Can you share your elasticsearch logs? We're looking for anything related to blocking writes. If there's no red flags there it's possible the index got put into a read only state for some other reason, you can reverse this by

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/.monitoring-*/_settings -d '{"index.blocks.read_only_allow_delete": null}'

https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html has more info on this reset.

3 Likes
6

Sorry i was impatient, so i have deleted all indexes, see if that resolve the issue, after deleting all and started creating new one. if the problem persists i will post the output here.

is there any way i can specifies index ( nodes) different custom folder.

from /var/lib/elasticsearch/nodes/ to /var/syslog-ng/lib/elasticsearch/nodes/ ?

appreciated your quick help

7

Yep, in elasticsearch.yml set path.data to /var/syslog-ng/lib/elasticsearch/nodes/. You'll probably want to stop elasticsearch, move your current data directory, and then reconfigure.

8

Thank you for your reply, i have solution for now, we can close this discussion.

In the future if i come across new issue i will open another one to discuss.

It is good product.

R!

9

No problem. If you get a chance can you share your solution?

10

Sure,

As i was mentioned, i have deleted the data and re-created, but if i see further issue i can follow your instruction to produce more logs, it will be helpfull if any bugs

Appreciate your help.

R!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.