Config: Error 403 Forbidden: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];


(R!) #1

Config: Error 403 Forbidden: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Kibana not able to retrieve old data.

I have enabled one of the device it started logging huge data, after some time Elastic Search and Kibana not showing data, i have restarted, but no luck, i see below errors

syslog-ng ---elasticksearch--kibana is my setup

[2017-12-27T20:07:44.841332] Outgoing message; message='Dec 27 20:07:44 ES6 kibana[7035]: {"type":"error","@timestamp":"2017-12-27T20:07:44Z","tags":["error","monitoring-ui"],"pid":7035,"level":"error","error":{"message":"[export_exception] Exception when closing export bulk","name":"Error","stack":"[export_exception] Exception when closing export bulk :: {"path":"/_xpack/monitoring/_bulk","query":{"system_id":"kibana","system_api_version":"6","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_settings\"}}\n{\"kibana_uuid\":\"65539add-f049-4084-800e-62ebe31172c5\",\"xpack\":{\"default_admin_email\":null}}\n{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"65539add-f049-4084-800e-62ebe31172c5\",\"name\":\"ES6\",\"index\":\".kibana\",\"host\":\"ES6\",\"transport_address\":\"192.168.1.75:5601\",\"version\":\"6.0.1\",\"snapshot\":false,\"status\":\"green\"},\"concurrent_connections\":393,\"os\":{\"load\":{\"1m\":0.03564453125,\"5m\":0.23828125,\"15m\":0.458984375},\"memory\":{\"total_in_bytes\":8371228672,\"free_in_bytes\":1713790976,\"used_in_bytes\":6657437696},\"uptime_in_millis\":1458957000},\"process\":{\"event_loop_delay\":97.87312602996826,\"memory\":{\"heap\":{\"total_in_bytes\":120438784,\"used_in_bytes\":102224128,\"size_limit\":1501560832},\"resident_set_size_in_bytes\":161980416},\"uptime_in_millis\":482334},\"requests\":{\"disconnects\":0,\"total\":205,\"status_codes\":{\"200\":109,\"304\":77,\"403\":14,\"404\":5}},\"response_times\":{\"average\":95,\"max\":699},\"timestamp\":\"2017-12-27T20:07:39.449Z\"}\n","statusCode":500,"response":"{\"took\":38,\"errors\":true,\"error\":{\"type\":\"export_exception\",\"reason\":\"Exception when closing export bulk\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulks\",\"caused_by\\x0a'
[2017-12-27T20:07:44.841386] Outgoing message; message='Dec 27 20:07:44 ES6 kibana[7035]: ":{\"type\":\"export_exception\",\"reason\":\"bulk [default_local] reports failures when exporting documents\",\"exceptions\":[{\"type\":\"export_exception\",\"reason\":\"ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused_by\":{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}},{\"type\":\"export_exception\",\"reason\":\"ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused_by\":{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}}]}}}}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:91:20)\n at IncomingMessage.emit (events.js:185:7)\n at endReadableNT (_stream_readable.js:974:12)\n at _combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickDomainCallback (internal/process/next_tick.js:128:9)"},"message":"[export_exception] Exception when closing export bulk"}\x0a'


(Jon Budzenski) #2

Can you verify that there's disk space available? ES will go into read only mode once a threshold is hit.


(R!) #3

Yes i have 2+GB available free, when i see that first i checked space.


(R!) #4

/dev/mapper/ES6--vg-root 12G 5.5G 5.3G 52% /


(Jon Budzenski) #5

Can you share your elasticsearch logs? We're looking for anything related to blocking writes. If there's no red flags there it's possible the index got put into a read only state for some other reason, you can reverse this by

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/.monitoring-*/_settings -d '{"index.blocks.read_only_allow_delete": null}'

https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html has more info on this reset.


(R!) #6

Sorry i was impatient, so i have deleted all indexes, see if that resolve the issue, after deleting all and started creating new one. if the problem persists i will post the output here.

is there any way i can specifies index ( nodes) different custom folder.

from /var/lib/elasticsearch/nodes/ to /var/syslog-ng/lib/elasticsearch/nodes/ ?

appreciated your quick help


(Jon Budzenski) #7

Yep, in elasticsearch.yml set path.data to /var/syslog-ng/lib/elasticsearch/nodes/. You'll probably want to stop elasticsearch, move your current data directory, and then reconfigure.


(R!) #8

Thank you for your reply, i have solution for now, we can close this discussion.

In the future if i come across new issue i will open another one to discuss.

It is good product.

R!


(Jon Budzenski) #9

No problem. If you get a chance can you share your solution?


(R!) #10

Sure,

As i was mentioned, i have deleted the data and re-created, but if i see further issue i can follow your instruction to produce more logs, it will be helpfull if any bugs

Appreciate your help.

R!


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.