Status 403 in Kibana POST request

[mortueta]

Hi all,

I am having some problems. To be honest since the upgrade from an older version to 6.5.3 I had some problems with kibana. But after index recreation it was solved. I started using spaces and until yesterday everything was working fine but today I tried to create some visualizations and I am getting a Forbidden message. Nothing has been changed.

Jan 15 09:23:58 hostname kibana[1226]: {"type":"response","@timestamp":"2019-01-15T08:23:58Z","tags":[],"pid":1226,"method":"post","statusCode":403,"req":{"url":"/api/saved_objects/visualization?overwrite=true","method":"post","headers":{"host":"SERVER-IP:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0","accept":"*/*","accept-language":"en,es-ES;q=0.8,es;q=0.5,en-US;q=0.3","accept-encoding":"gzip, deflate","referer":"http://SERVER-IP:5601/s/security-operations/app/kibana","content-type":"application/json","kbn-version":"6.5.3","origin":"http://SERVER-IP:5601","content-length":"792","connection":"keep-alive"},"remoteAddress":"CLIENT-IP","userAgent":"CLIENT-IP","referer":"http://SERVER-IP:5601/s/security-operations/app/kibana"},"res":{"statusCode":403,"responseTime":70,"contentLength":9},"message":"POST /api/saved_objects/visualization?overwrite=true 403 70ms - 9.0B"}

Some info regarding installation

• ElasticSearch 6.5.3
• Logstash 6.5.3
• Kibana 6.5.3
• NO X-PACK security
• Plugins: ingest-geoip, ingest-user-agent

{
  ".kibana" : {
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "read_only_allow_delete" : "false"
        },
        "provided_name" : ".kibana",
        "creation_date" : "1541758243947",
        "number_of_replicas" : "0",
        "uuid" : "h_ekz5tNRqWaP4I1OzjGmQ",
        "version" : {
          "created" : "6040399",
          "upgraded" : "6050399"
        }
      }
    }
  }
}

Thank you in advance!

[BenH]

We're actually seeing something similar in a new cluster I setup on Friday.

Everything was humming along great over the weekend until some point Monday the 14th AM and now see the same Forbidden errors you are getting.

[timroes]

@Larry_Gregory could you help here how to debug that issue properly?

[Larry_Gregory]

Since you're running without x-pack security, it is likely that the 403/Forbidden error is coming from Elasticsearch itself.

Can you open your browser's dev tools (instructions here) and inspect the response when one of these errors happens? It will likely include more information to help us troubleshoot. In my example below, the Kibana index is set to read-only, which mimics the error your'e seeing. Your index settings don't mention it being read-only, so it might be a different issue for you:

[BenH]

I have managed to get everything back up and running.

For us, I believe the system locked up when it ran out of available HDD space. The solution was as simple as increasing their volume size and rebooting and it began to write logs once more.

[mortueta]

Hi,

First of all, thanks to the ElasticTeam: @timroes for addressing my problem and to @Larry_Gregory for the support.

The thing is that I wasn't getting the settings of the right .kibana index. In the index used by kibana the pararameter "read_only_allow_delete" was set to : "true"

[quote="mortueta, post:1, topic:164267"]
had some problems with kibana. But after index recreation it was solved.
[/quote] To solve this problem in kibana.yml I set kibana.index: ".newkibana". Now I have these system indexes:

How do the Kibana indices work within ELK?

@BenH , it happened the same to me, I ran out off disk and the "read_only_allow_delete" was set to : "true". I set it again to false in the correct index and it is working fine!

Solution

Put to false the correct kibana index. IE:

PUT /.newkibana_1/_settings
{
"index.blocks.read_only_allow_delete": false
}

Kind regards,

M

[Larry_Gregory]

@mortueta / @BenH

Awesome, glad that solved your problem!

How do the Kibana indices work?

Kibana introduced the concept of "Saved Object Migrations" in version 6.5, which allows saved objects (dashboards, index patterns, etc) to update their internal schema with new versions of Kibana. In order to accomplish this, Kibana creates indices as needed, and a single alias.

The alias is set to the value of kibana.index -- usually .kibana. This alias will point to one of the .kibana_N indices, where N is the most recent version of the kibana index. This is an implementation detail though, and shouldn't matter for end users. (@christophilus please correct me if I misrepresented anything here)