Error saving Spaces


(Roger) #1

Hi all,
I'm trying to create (using Kibana UI) a Spaces on Kibana 6.5 but I'm facing the error "Error saving space:".
so I tried to create it using API. every command ends correctly but when I try to search it using Kibana UI I cannot find anything.

this is the output of GET /api/spaces/space

{
"_index" : "api",
"_type" : "spaces",
"_id" : "space",
"_version" : 1,
"found" : true,
"_source" : {
"id" : "marketing",
"name" : "Marketing",
"description" : "This is the Marketing Space",
"color" : "#aabbcc",
"initials" : "MK"
}
}


(Brandon Kobel) #2

Hey @rschirin, is that the full error that you're seeing when trying to create the Space using the UI? If you check your Kibana logs, are there any additional errors shown there?

Also, how are you trying to search for the Space in the UI, are you not seeing it listed in the Space Management screen?


(Roger) #3

yes, it is the full error. so, it is empty.
if I take a look to kibana.out I can see this:

{"type":"response","@timestamp":"2018-11-16T18:26:41Z","tags":,"pid":26008,"method":"post","statusCode":403,"req":{"url":"/api/spaces/space","method":"post","headers":{"host":"saelk2.domain.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0","accept":"application/json, text/plain, /","accept-language":"it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://saelk2.domain.com:5601/app/kibana","content-type":"application/json;charset=utf-8","kbn-version":"6.5.0","content-length":"37","connection":"keep-alive"},"remoteAddress":"10.0.37.149","userAgent":"10.0.37.149","referer":"http://saelk2.domain.com:5601/app/kibana"},"res":{"statusCode":403,"responseTime":14,"contentLength":9},"message":"POST /api/spaces/space 403 14ms - 9.0B"}

no, I cannot see it in the Space Management screen :persevere:


(Brandon Kobel) #4

Hey @rschirin, are you using Kibana with security? If so, what user are you using and which roles are assigned?


(Roger) #5

these are the setted keys in my kibana.yml :

server.port: 5601
server.host: 0.0.0.0

elasticsearch.url: "https://localhost:9200"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "xxx"
xpack.reporting.encryptionKey: "xxx"
pid.file: /var/run/kibana/kibana.pid

I have a gold subscription. nothing else....


(Brandon Kobel) #6

When you access Kibana, are you being prompted to login? If you are, would you mind running the following query using DevTools and replying with the response? GET _xpack/security/_authenticate

The reason I'm asking is that you're getting a 403 when you try to save the Space, which is generally when you're using security and you don't have the necessary privileges.


(Roger) #7

sorry since I'm late, anyway this is the output result:

{
"username" : "roger",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"ldap_dn" : "CN=roger,OU=User Accounts,DC=dc,DC=dc,DC=com",
"ldap_groups" : [
"a lot of them"
]
},
"enabled" : true
}

so, it seems ok.


#8

Hello,
I've noticed the same behavior since the ELK update from 6.4.2 to 6.5. Like described in the OP I can create spaces through API requests but in the UI I get "Error Saving Space:".

An similar behavior I can observe when I try to manage a security Role. After clicking on a Role the Kibana status bar is showing up, but the UI keeps staying in the role lists. The kibana.log shows up an GET againts /api/spaces/space with 403 as response.

here the relevant log entries:
https://pastebin.com/TgrHV2Cz

GET _xpack/security/_authenticate:

Summary

{
"username" : "elastic",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}


(Brandon Kobel) #9

@Alufolie, it looks like the call to retrieve a list of Spaces is throwing a 403 for you. If you go to Management -> Spaces, do you see any Spaces listed?


(Brandon Kobel) #10

@rschirin interesting, you have the superuser role, so you shouldn't be getting a 403 when you try to save the Space. If you open up your browser's developer tools, and look at the network requests that are being made, do you see a specific request getting the 403?


#11

Hey @Brandon_Kobel, thank you for the answer!

No I cant see any Spaces. Neither the created space through the api nor the default space.
So, same behavior like described in the OP.

The Kibana UI gives me also a 403 when I try to add a space through management -> spaces while monitoring the traffic with the browser dev tools.


(Brandon Kobel) #12

@Alufolie, have you specified the kibana.index setting in your kibana.yml?


#13

@Brandon_Kobel
yes.
kibana.index: ".kibana"


(Roger) #14

@Brandon_Kobel, opening Management-->Spaces section I'm getting this:

GET https://ftpmonitoring.crifnet.com/api/spaces/space 403 @ vendors.bundle.js:277

but I'm not facing any popup error. it seems to be hidden.
then, when I click on Create Space (after that I filled a name) I faced this one:

POST https://ftpmonitoring.crifnet.com/api/spaces/space 403 @ vendors.bundle.js:277

so, effectively there is a 403 error


(Brandon Kobel) #15

I'm having trouble replicating what you all are seeing. You you all mind running the following query using the Dev Tools application in Kibana and replying with the response?

GET _xpack/security/user/_privileges

(Roger) #16

that's the output:

{
  "cluster" : [
    "all"
  ],
  "global" : [ ],
  "indices" : [
    {
      "names" : [
        "*"
      ],
      "privileges" : [
        "all"
      ]
    }
  ],
  "applications" : [
    {
      "application" : "*",
      "privileges" : [
        "*"
      ],
      "resources" : [
        "*"
      ]
    }
  ],
  "run_as" : [
    "*"
  ]
}

(Brandon Kobel) #17

Thanks for working through this with me @rschirin, I really appreciate it.

Those privileges look completely fine for the superuser role which I saw assigned.

Would you mind also running the following two queries using Dev Tools and replying with their responses?

GET .kibana/doc/_search
{
  "query": {
        "terms" : { "type" : ["space"]}
    }
}
GET _xpack/security/user/_has_privileges
{
  "applications": [
    {
      "application": "kibana-.kibana",
      "resources": "space:default",
      "privileges": [
        "action:login"
      ]
    }
  ]
}

(Roger) #18

hey @Brandon_Kobel, are you kidding me? thank you for the help

this is the output of the first query:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".kibana_7",
        "_type" : "doc",
        "_id" : "space:default",
        "_score" : 1.0,
        "_source" : {
          "space" : {
            "name" : "Default",
            "description" : "This is your default space!",
            "color" : "#00bfb3",
            "_reserved" : true
          },
          "type" : "space",
          "updated_at" : "2018-11-16T17:14:11.779Z"
        }
      }
    ]
  }
}

while this is the output of the second query:

{
  "username" : "l72rugschiri",
  "has_all_requested" : true,
  "cluster" : { },
  "index" : { },
  "application" : {
    "kibana-.kibana" : {
      "space:default" : {
        "action:login" : true
      }
    }
  }
}

(Brandon Kobel) #19

@rschirin those responses make me even more confused, as you should be being authorized to list the Spaces then...

If you could set the following two settings in your kibana.yml, this will enable our security audit logs and write the log output to a file:

xpack.security.audit.enabled: true
logging.dest: /var/log/kibana.log

if you could then restart Kibana, go to Management -> Spaces to try to list all of the Spaces, and then attach the log output here, hopefully that'll provide some insight into what is going awry.

Also, would you mind sharing your full kibana.yml here (redacting the sensitive settings)?


(Roger) #20

so, enabling audit key I saw this error in the kibana.out file:

{"type":"response","@timestamp":"2018-11-21T11:29:37Z","tags":[],"pid":16301,"method":"post","statusCode":403,"req":{"url":"/api/spaces/space","method":"post","headers":{"host":"saelk1.net.com:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0","accept":"application/json, text/plain, */*","accept-language":"it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://saelk1.net.com:5601/app/kibana","content-type":"application/json;charset=utf-8","kbn-version":"6.5.0","content-length":"27","connection":"keep-alive"},"remoteAddress":"10.0.37.149","userAgent":"10.0.37.149","referer":"http://saelk1.net.com:5601/app/kibana"},"res":{"statusCode":403,"responseTime":37,"contentLength":9},"message":"POST /api/spaces/space 403 37ms - 9.0B"}

here you can find my kibana.yml:

server.port: 5601
server.host: 0.0.0.0
elasticsearch.url: "https://localhost:9200"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "xxx"
xpack.reporting.encryptionKey: "xxxxxx"
xpack.infra.sources.default.metricAlias: "m.*"
elasticsearch.ssl.verificationMode: none
pid.file: /var/run/kibana/kibana.pid
logging.dest: /var/log/kibana/kibana.out
xpack.security.audit.enabled: true