Vagrant Install of Kibana works, Vagrant install of Kibana + X-Pack doesn't work unless you explicitly declare elasticsearch.requestHeadersWhitelist: ['authorization']

I'm using Vagrant to test out some stuff with x-pack. I spin up my brand new ES, LS and Kibana 5.3.0 instance (everything on the same machine) and everything works as expected. I can reach Kibana, and ES normally

The issue is that when I install X-Pack on Kibana, X-Pack installs without any issues. Kibana shows the login form at http://localhost:5601/login?next=%2Fapp%2Fkibana#?_g=() as expected.

According to the install documentation , I should just be able to enter the L/P and get into Kibana to begin working with the X-Pack stuff

When I enter the default user and password (kibana or elastic/changeme), Kibana throws 401 errors all the time, every time.

{"statusCode":401,"error":"Unauthorized","message":"[security_exception] missing authentication token for REST request [/_xpack/security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack/security/_authenticate\",\"query\":{},\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_xpack/security/_authenticate]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_xpack/security/_authenticate]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

I've also used Firefox, where I've added the Authentication header and the username/password hash to the POST and I still get the error.

Request
Host: localhost:5601
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: application/json, text/plain, /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=utf-8
kbn-version: 5.3.0
Referer: http://localhost:5601/login?next=%2Fapp%2Fkibana
Content-Length: 44
Cookie: _ga=GA1.1.1402432993.1453231728
Connection: keep-alive
Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==
Pragma: no-cache
Cache-Control: no-cache

Response
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Date: Tue, 11 Apr 2017 20:05:48 GMT
Set-Cookie: sid=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; HttpOnly; Path=/
Transfer-Encoding: chunked
Vary: accept-encoding
kbn-name: kibana
kbn-version: 5.3.0
kbn-xpack-sig: 2ca3dcb0a4b1eda3162b117e55246ab1

The kicker is that Elasticsearch works as expected, I add Basic Auth with the default elastic/changeme username/password combination to Postman and

localhost:9200/_xpack/

{
  "build": {
    "hash": "f14da62",
    "date": "2017-03-23T03:59:37.018Z"
  },
  "license": {
    "uid": "f788db41-2f17-45a5-980e-21614d4b596c",
    "type": "trial",
    "mode": "trial",
    "status": "active",
    "expiry_date_in_millis": 1494529614174
  },
  "features": {
    "graph": {
      "description": "Graph Data Exploration for the Elastic Stack",
      "available": true,
      "enabled": true
    },
    "monitoring": {
      "description": "Monitoring for the Elastic Stack",
      "available": true,
      "enabled": true
    },
    "security": {
      "description": "Security for the Elastic Stack",
      "available": true,
      "enabled": true
    },
    "watcher": {
      "description": "Alerting, Notification and Automation for the Elastic Stack",
      "available": true,
      "enabled": true
    }
  },
  "tagline": "You know, for X"
}

Vagrant file:
Vagrant.configure('2') do |config|
config.vm.box = 'ubuntu/trusty64'
config.vm.hostname = 'elasticsearch'

  config.vm.provider :virtualbox do |v, override|
    v.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
    v.customize ['modifyvm', :id, '--natdnsproxy1', 'on']
    v.customize ['modifyvm', :id, "--memory", 4096]
    v.customize ['modifyvm', :id, '--name', 'elasticsearch']
  end

  config.vm.network 'private_network', ip: "10.0.2.15"

  #Elasticsearch
  config.vm.network :forwarded_port, guest: 9200, host: 9200, protocol: 'tcp'  # REST
  config.vm.network :forwarded_port, guest: 9300, host: 9300, protocol: 'tcp'  # Java connector

  #Kibana
  config.vm.network :forwarded_port, guest: 5601, host: 5601, protocol: 'tcp'  # Kibana

  config.ssh.forward_agent = true

end

Why would this work without X-Pack then stop working with it?

This is VERY similar to how our full-stack integration tests work. They spin up a Ubuntu (or other) VM with everything installed on it, and then run Selenium UI tests against Kibana from the host machine. So it should work. I tested the 5.3.0 release this way.

You do have a few differences in your network configuration and port mapping.

Here's my Vagrantfile;

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  #config.vm.box = "elastic/ubuntu-14.04-x86_64"

  config.vm.network :forwarded_port, guest:5601, host:5601
  config.vm.network :forwarded_port, guest:9200, host:9200

  config.ssh.insert_key = false

  config.vm.provision "shell", path: "qa/provision.sh"

  config.vm.provider :virtualbox do |vb|
    vb.memory = 6144
    vb.cpus = 4
    vb.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate//vagrant","1"]
  end

end

I guess when I don't specify the protocol: 'tcp' I get both tcp and udp. Could be something there. Easy to test the change.
You have some other nat settings, but nothing that looks like a problem.

I set a larger memory size because I also install logstash and 3 beats. But yours should be OK.

The only other thing to try might be a new incognito browser window. If you've run different versions of Kibana it can leave some conflicting stuff in the browser cache that could cause this type of problem.

Regards,
Lee

As an update to this:

This is NOT a Virtualbox issue, and I've updated the title accordingly

The Ansible role running in Vagrant accidentally passed in the

elasticsearch.requestHeadersWhitelist: ['authorization']

as

elasticsearch.requestHeadersWhitelist: {}

which caused the unauthorized issues I described above.

So, if you are always getting:

{"statusCode":401,"error":"Unauthorized","message":"[security_exception] missing authentication token for REST request [/_xpack/security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack/security/_authenticate\",\"query\":{},\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_xpack/security/_authenticate]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_xpack/security/_authenticate]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

errors. Ensure elasticsearch.requestHeadersWhitelist: ['authorization'] is explicitly set.

I'm glad you got things working. And thanks for updating this post for others to find.

I'm afraid I don't understand this issue completely because mine is working without setting `elasticsearch.requestHeadersWhitelist: ['authorization']

Did you have the requestHeadersWhitelist from the very start? Or did you add it after some initial problems? If you added it from the start, but Ansible caused the parameter value to be wrong, maybe you really didn't need it at all?

Just to be clear, since I map port 5601 from the guest VM to the same port on the host, I can open and log in to Kibana from the host or guest (when there's a desktop) with http://localhost:5601 (or https://localhost:5601 when I have ssl configured).
I think you're accessing it the same way based on your
Request
Host: localhost:5601

Again, we have a few differences in our VM network configuration so maybe that is why you have to set the whitelist?
I've used by same VM, Elasticsearch, Kibana configuration with Ubuntu 14.04, CentOS-7, and a Ubuntu 16.04 with Desktop and haven't had to set the whitelist.

I'll ask some Kibana developers to read through this and see if they can shed some light on it.

Regards,
Lee

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.