Error saving Spaces

rschirin · November 16, 2018, 6:16pm

Hi all,
I'm trying to create (using Kibana UI) a Spaces on Kibana 6.5 but I'm facing the error "Error saving space:".
so I tried to create it using API. every command ends correctly but when I try to search it using Kibana UI I cannot find anything.

this is the output of GET /api/spaces/space

{
"_index" : "api",
"_type" : "spaces",
"_id" : "space",
"_version" : 1,
"found" : true,
"_source" : {
"id" : "marketing",
"name" : "Marketing",
"description" : "This is the Marketing Space",
"color" : "#aabbcc",
"initials" : "MK"
}
}

Brandon_Kobel · November 16, 2018, 6:25pm

Hey @rschirin, is that the full error that you're seeing when trying to create the Space using the UI? If you check your Kibana logs, are there any additional errors shown there?

Also, how are you trying to search for the Space in the UI, are you not seeing it listed in the Space Management screen?

rschirin · November 16, 2018, 6:28pm

yes, it is the full error. so, it is empty.
if I take a look to kibana.out I can see this:

{"type":"response","@timestamp":"2018-11-16T18:26:41Z","tags":,"pid":26008,"method":"post","statusCode":403,"req":{"url":"/api/spaces/space","method":"post","headers":{"host":"saelk2.domain.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0","accept":"application/json, text/plain, /","accept-language":"it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://saelk2.domain.com:5601/app/kibana","content-type":"application/json;charset=utf-8","kbn-version":"6.5.0","content-length":"37","connection":"keep-alive"},"remoteAddress":"10.0.37.149","userAgent":"10.0.37.149","referer":"http://saelk2.domain.com:5601/app/kibana"},"res":{"statusCode":403,"responseTime":14,"contentLength":9},"message":"POST /api/spaces/space 403 14ms - 9.0B"}

no, I cannot see it in the Space Management screen

Brandon_Kobel · November 16, 2018, 6:39pm

Hey @rschirin, are you using Kibana with security? If so, what user are you using and which roles are assigned?

rschirin · November 16, 2018, 6:47pm

these are the setted keys in my kibana.yml :

server.port: 5601
server.host: 0.0.0.0

elasticsearch.url: "https://localhost:9200"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "xxx"
xpack.reporting.encryptionKey: "xxx"
pid.file: /var/run/kibana/kibana.pid

I have a gold subscription. nothing else....

Brandon_Kobel · November 16, 2018, 7:19pm

When you access Kibana, are you being prompted to login? If you are, would you mind running the following query using DevTools and replying with the response? GET _xpack/security/_authenticate

The reason I'm asking is that you're getting a 403 when you try to save the Space, which is generally when you're using security and you don't have the necessary privileges.

rschirin · November 19, 2018, 9:15am

sorry since I'm late, anyway this is the output result:

{
"username" : "roger",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"ldap_dn" : "CN=roger,OU=User Accounts,DC=dc,DC=dc,DC=com",
"ldap_groups" : [
"a lot of them"
]
},
"enabled" : true
}

so, it seems ok.

Alufolie · November 19, 2018, 10:14am

Hello,
I've noticed the same behavior since the ELK update from 6.4.2 to 6.5. Like described in the OP I can create spaces through API requests but in the UI I get "Error Saving Space:".

An similar behavior I can observe when I try to manage a security Role. After clicking on a Role the Kibana status bar is showing up, but the UI keeps staying in the role lists. The kibana.log shows up an GET againts /api/spaces/space with 403 as response.

here the relevant log entries:
https://pastebin.com/TgrHV2Cz

GET _xpack/security/_authenticate:

Summary

{
"username" : "elastic",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}

Brandon_Kobel · November 19, 2018, 3:36pm

@Alufolie, it looks like the call to retrieve a list of Spaces is throwing a 403 for you. If you go to Management -> Spaces, do you see any Spaces listed?

Brandon_Kobel · November 19, 2018, 3:38pm

@rschirin interesting, you have the superuser role, so you shouldn't be getting a 403 when you try to save the Space. If you open up your browser's developer tools, and look at the network requests that are being made, do you see a specific request getting the 403?

Alufolie · November 19, 2018, 4:10pm

Hey @Brandon_Kobel, thank you for the answer!

No I cant see any Spaces. Neither the created space through the api nor the default space.
So, same behavior like described in the OP.

The Kibana UI gives me also a 403 when I try to add a space through management -> spaces while monitoring the traffic with the browser dev tools.

Brandon_Kobel · November 19, 2018, 4:45pm

@Alufolie, have you specified the kibana.index setting in your kibana.yml?

Alufolie · November 20, 2018, 7:33am

@Brandon_Kobel
yes.
kibana.index: ".kibana"

rschirin · November 20, 2018, 12:45pm

@Brandon_Kobel, opening Management-->Spaces section I'm getting this:

GET https://ftpmonitoring.crifnet.com/api/spaces/space 403 @ vendors.bundle.js:277

but I'm not facing any popup error. it seems to be hidden.
then, when I click on Create Space (after that I filled a name) I faced this one:

POST https://ftpmonitoring.crifnet.com/api/spaces/space 403 @ vendors.bundle.js:277

so, effectively there is a 403 error

Brandon_Kobel · November 20, 2018, 3:23pm

I'm having trouble replicating what you all are seeing. You you all mind running the following query using the Dev Tools application in Kibana and replying with the response?

GET _xpack/security/user/_privileges

rschirin · November 20, 2018, 4:12pm

that's the output:

{
  "cluster" : [
    "all"
  ],
  "global" : [ ],
  "indices" : [
    {
      "names" : [
        "*"
      ],
      "privileges" : [
        "all"
      ]
    }
  ],
  "applications" : [
    {
      "application" : "*",
      "privileges" : [
        "*"
      ],
      "resources" : [
        "*"
      ]
    }
  ],
  "run_as" : [
    "*"
  ]
}

Brandon_Kobel · November 20, 2018, 4:47pm

Thanks for working through this with me @rschirin, I really appreciate it.

Those privileges look completely fine for the superuser role which I saw assigned.

Would you mind also running the following two queries using Dev Tools and replying with their responses?

GET .kibana/doc/_search
{
  "query": {
        "terms" : { "type" : ["space"]}
    }
}

GET _xpack/security/user/_has_privileges
{
  "applications": [
    {
      "application": "kibana-.kibana",
      "resources": "space:default",
      "privileges": [
        "action:login"
      ]
    }
  ]
}

rschirin · November 20, 2018, 5:25pm

hey @Brandon_Kobel, are you kidding me? thank you for the help

this is the output of the first query:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".kibana_7",
        "_type" : "doc",
        "_id" : "space:default",
        "_score" : 1.0,
        "_source" : {
          "space" : {
            "name" : "Default",
            "description" : "This is your default space!",
            "color" : "#00bfb3",
            "_reserved" : true
          },
          "type" : "space",
          "updated_at" : "2018-11-16T17:14:11.779Z"
        }
      }
    ]
  }
}

while this is the output of the second query:

{
  "username" : "l72rugschiri",
  "has_all_requested" : true,
  "cluster" : { },
  "index" : { },
  "application" : {
    "kibana-.kibana" : {
      "space:default" : {
        "action:login" : true
      }
    }
  }
}

Brandon_Kobel · November 20, 2018, 6:31pm

@rschirin those responses make me even more confused, as you should be being authorized to list the Spaces then...

If you could set the following two settings in your kibana.yml, this will enable our security audit logs and write the log output to a file:

xpack.security.audit.enabled: true
logging.dest: /var/log/kibana.log

if you could then restart Kibana, go to Management -> Spaces to try to list all of the Spaces, and then attach the log output here, hopefully that'll provide some insight into what is going awry.

Also, would you mind sharing your full kibana.yml here (redacting the sensitive settings)?

rschirin · November 21, 2018, 11:37am

so, enabling audit key I saw this error in the kibana.out file:

{"type":"response","@timestamp":"2018-11-21T11:29:37Z","tags":[],"pid":16301,"method":"post","statusCode":403,"req":{"url":"/api/spaces/space","method":"post","headers":{"host":"saelk1.net.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0","accept":"application/json, text/plain, */*","accept-language":"it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://saelk1.net.com:5601/app/kibana","content-type":"application/json;charset=utf-8","kbn-version":"6.5.0","content-length":"27","connection":"keep-alive"},"remoteAddress":"10.0.37.149","userAgent":"10.0.37.149","referer":"http://saelk1.net.com:5601/app/kibana"},"res":{"statusCode":403,"responseTime":37,"contentLength":9},"message":"POST /api/spaces/space 403 37ms - 9.0B"}

here you can find my kibana.yml:

server.port: 5601
server.host: 0.0.0.0
elasticsearch.url: "https://localhost:9200"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "xxx"
xpack.reporting.encryptionKey: "xxxxxx"
xpack.infra.sources.default.metricAlias: "m.*"
elasticsearch.ssl.verificationMode: none
pid.file: /var/run/kibana/kibana.pid
logging.dest: /var/log/kibana/kibana.out
xpack.security.audit.enabled: true