In my home lab, I am running 6.2.2 of Elasticsearch, Logstash and Kibana all on a single machine.
I really only had three logs or so going into it, pfsense, windows, and bro and three index patterns. But after I added a fourth index, I found I could not create a new index pattern. I tried to create it, but nothing happened. No errors pop up, nothing in the kibana.stderr log, nothing.
After some looking around, I found that all the indices were yellow and the logstash log was full of this error:
[2018-11-22T13:27:52,557][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
After checking my logstash conf and comparing the indexes I was sending to ES and what was currently in ES I realized that half the logs in ES were no longer being used and those were the logs that were the basis of my current index patterns. So I decided to delete the old logs that were not in use and then remove the index-patterns and create new patterns for my current indices.
I then did a GET _cat/indices on the dev tools page and found that all my indices were yellow. When I checked on the health of the cluster, I had a lot of unassigned shards. By doing an GET _cluster/allocation/explain on the dev tools page, I found that ES was complaining about allocating shards to the same node (which is verboten). So since this is a home lab, I set the replicas to 0 for the indices I wanted to keep.
I then deleted the other indices I no longer needed.
At this point I continued to run into problems.
After deleting the indices, I had three index patterns I needed to delete:
pfsense2-2018
windows-2018
broids2-2018
On Management - Index Pattern I clicked on Refresh icon on top right I got an error screen with the following text:
blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
Error: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at http://192.168.1.101:5601/bundles/commons.bundle.js?v=16588:1:293164
at processQueue (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:132456)
at http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:133349
at Scope.$digest (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:144239)
at Scope.$apply (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:147018)
at done (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:100026)
at completeRequest (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:104697)
at XMLHttpRequest.xhr.onload (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:105435)
I then tried clicking on the trash can icon to delete index pattern. This time I got the error screen with the following text:
OOPS! Looks like something went wrong, Refreshing may do the trick.
Fatal Error
blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
Version: 6.2.2
Build: 16588
Error: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at http://192.168.1.101:5601/bundles/commons.bundle.js?v=16588:1:293164
at processQueue (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:132456)
at http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:133349
at Scope.$digest (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:144239)
at Scope.$apply (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:147018)
at done (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:100026)
at completeRequest (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:104697)
at XMLHttpRequest.xhr.onload (http://192.168.1.101:5601/bundles/vendors.bundle.js?v=16588:58:105435)
Based on some of the suggestions from the references below, I went to find out if the index pattern was in the .kibana index. So from dev tools to find out information about the index-pattern, I ran: GET .kibana/index-pattern/windows-2018
{
"_index": ".kibana",
"_type": "index-pattern",
"_id": "windows-2018",
"found": false
}
Same result from command line:
curl -Xget "http://localhost:9200/.kibana/index-pattern/windows-2018"
{"_index":".kibana","_type":"index-pattern","_id":"windows-2018","found":false}
Which meant kibana thought the index-pattern was gone, but in the drop down for the discover tab and on the management - index patter page, they were still listed.
I restarted kibana and the patterns were still there but the indices are not. For reference, here are results of GET _cat/indices:
green open .kibana 72xu3JySRXOCUvjRAxsHMg 1 0 5 2 89.1kb 89.1kb
green open nxlog Dmj4Mk8zQJmv02ueoGsT0Q 5 0 368 0 630.2kb 630.2kb
green open broids-2018 P0fI96T8Q-GxTjX4uQq4aA 5 0 926248 0 1.1gb 1.1gb
green open pfsense3-2018 b4Cn-rCFScSxYjL3gNaCwg 5 0 221315 0 83.3mb 83.3mb
Tried to delete the other index patterns via the gui and got the same errors.
tried to delete via the command line:
curl -XDELETE "http://localhost:5601/api/saved_objects/index-pattern/pfsense2-2018" -H 'kbn-xsrf: true'
{"message":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];","statusCode":403,"error":"Forbidden"}
I tried this also:
curl -XDELETE "http://localhost:9200/.kibana/index-pattern/pattern_name"
Same read-only error.
I also still cannot add an index pattern.
Now what? Is this a bug or am I missing something? Thanks.
Some of the research I've done on this issue: