Kibana xpack missing authentication token for REST request


(Colin) #1

I'm getting peppered with this message pretty much non-stop and haven't been able to figure out what's causing my issue, could someone point me in the right direction? I believe this is causing my overall problem, which is that kibana is reporting an Authentication Exception under my status breakdowns under the monitoring plugin, and as such the status is red.

I'm finding the error in my kibana.log, here is the full message:

{"type":"log","@timestamp":"2017-09-20T21:27:06Z","tags":
["license","warning","xpack"],"pid":38255,"message":"License information could not be obtained from Elasticsearch. 
[security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-
Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: 
{\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":
[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request 
[/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-
8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request 
[/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-
8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-
8\\\"\"}"}

This is my kibana.yml:

# ---------------------------------- Server -----------------------------------
server:
  port: 5601
  host: "localhost"
  #basePath: ""
  maxPayloadBytes: 1048576
  name: "hyperv-kibana"
  ssl:
    #enabled: false
    #certificate: /path/to/your/server.crt
    #key: /path/to/your/server.key

pid.file: /var/run/kibana.pid
ops.interval: 5000

# ------------------------------- Elasticsearch --------------------------------
elasticsearch:
  url: "http://localhost:9200"
  preserveHost: true
  username: "elastic"
  password: "changeme"
  #ssl:
    #certificate: /path/to/your/client.crt
    #key: /path/to/your/client.key
    #certificateAuthorities: [ "/path/to/your/CA.pem" ]
    #verificationMode: full
  pingTimeout: 1500
  requestTimeout: 30000
  shardTimeout: 0
  startupTimeout: 5000
  #requestHeadersWhitelist: [ authorization ]
  #customHeaders: {}

# ---------------------------------- Kibana ------------------------------------
kibana:
  index: ".kibana"
  defaultAppId: "discover"

# ---------------------------------- X-Pack ------------------------------------
xpack:
  security.enabled: true
  monitoring:
    enabled: true
    max_bucket_size: 10000
    min_interval_seconds: 10000
    node_resolver: "uuid"
    report_stats: false
    kibana.collection.enabled: true
    kibana.collection.interval: 10000
  graph.enabled: true
  watcher.enabled: true
  reporting.enabled: true

# ---------------------------------- Logging -----------------------------------
logging:
  dest: /var/log/kibana/
  silent: false
  quiet: false
  verbose: true

(Tim Sullivan) #2

Which version of Kibana is this?

Is there a proxy in between Kibana and Elasticsearch?

Looks like you have verbose logging turned on. Does the Kibana log show anything logged from the Monitoring plugin about what it's doing?


(Colin) #3

Thanks Tsullivan, here is the version information:

kibana-5.6.1-1.x86_64
x-pack@5.6.1

elasticsearch-5.6.1-1.noarch

Name: x-pack
Description: Elasticsearch Expanded Pack Plugin
Version: 5.6.1
Native Controller: true
 * Classname: org.elasticsearch.xpack.XPackPlugin

There is no proxy, everything is installed on a single machine.

These logs are returned in the kibana.stderr output:

Elasticsearch ERROR: 2017-09-20T20:59:49Z
  Error: Request error, retrying
  GET http://localhost:9200/_xpack/security/_authenticate => connect ECONNREFUSED 127.0.0.1:9200
      at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:246:18)
      at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)
      at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at Socket.socketErrorListener (_http_client.js:309:9)
      at emitOne (events.js:96:13)
      at Socket.emit (events.js:188:7)
      at emitErrorNT (net.js:1277:8)
      at _combinedTickCallback (internal/process/next_tick.js:80:11)
      at process._tickDomainCallback (internal/process/next_tick.js:128:9)

Elasticsearch ERROR: 2017-09-20T20:59:49Z
  Error: Request error, retrying
  GET http://localhost:9200/_xpack/security/_authenticate => connect ECONNREFUSED 127.0.0.1:9200
      at Log.error (/usr/share/kibana/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:246:18)
      at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)
      at ClientRequest.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at Socket.socketErrorListener (_http_client.js:309:9)
      at emitOne (events.js:96:13)
      at Socket.emit (events.js:188:7)
      at emitErrorNT (net.js:1277:8)
      at _combinedTickCallback (internal/process/next_tick.js:80:11)
      at process._tickDomainCallback (internal/process/next_tick.js:128:9)

Elasticsearch WARNING: 2017-09-20T20:59:49Z
  Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-09-20T20:59:49Z
  No living connections

Elasticsearch WARNING: 2017-09-20T20:59:49Z
  Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-09-20T20:59:49Z
  No living connections

Elasticsearch WARNING: 2017-09-20T20:59:49Z
  Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-09-20T20:59:49Z
  No living connections

There is nothing in the stdout log.

Thanks


(Tim Sullivan) #4

Can you confirm that Elasticsearch is running by doing:

curl -u elastic:changeme http://localhost:9200

And the availability of X-Pack info to Kibana by doing:

curl -u kibana:changeme http://localhost:9200/_xpack

Note to change the passwords for the elastic user and kibana user to what yours are set to.


(Colin) #5
[root@localhost etc]# curl -u elastic:changeme http://localhost:9200
{
  "name" : "node1",
  "cluster_name" : "hyperv",
  "cluster_uuid" : "3XG5tAjPSEqoadjhZHreOg",
  "version" : {
    "number" : "5.6.1",
    "build_hash" : "667b497",
    "build_date" : "2017-09-14T19:22:05.189Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

and

[root@localhost etc]# curl -u kibana:changeme http://localhost:9200/_xpack?pretty
{
  "build" : {
    "hash" : "57f6e53",
    "date" : "2017-09-14T19:39:17.453Z"
  },
  "license" : {
    "uid" : "8cedae9a-b326-4255-b7ef-c2166e0bc6d2",
    "type" : "trial",
    "mode" : "trial",
    "status" : "active",
    "expiry_date_in_millis" : 1508456574120
  },
  "features" : {
    "graph" : {
      "description" : "Graph Data Exploration for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "ml" : {
      "description" : "Machine Learning for the Elastic Stack",
      "available" : true,
      "enabled" : true,
      "native_code_info" : {
        "version" : "5.6.1",
        "build_hash" : "e81fa9e3f3e0c1"
      }
    },
    "monitoring" : {
      "description" : "Monitoring for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "security" : {
      "description" : "Security for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "watcher" : {
      "description" : "Alerting, Notification and Automation for the Elastic Stack",
      "available" : true,
      "enabled" : true
    }
  },
  "tagline" : "You know, for X"
}

(Tim Sullivan) #6

Is Kibana actively logging that it can't reach Elasticsearch? Does it clear up if you restart Kibana?


(Colin) #7

Restarting does not resolve the issue; I'm getting the log I posted in the first comment periodically. The main issue is that I'm getting a message when I log into the kibana interface saying there is an authentication exception, and as such am not able to use the tool at all.


(Tim Sullivan) #8

Sounds it's just Monitoring reporting the red status. Does the issue go away if you disable Monitoring (xpack.monitoring.enabled: false in kibana.yml) and restart Kibana?


(André Ramos) #9

@tsullivan

Sorry but I have the same issue.
[igs@sligslogs01 ~]$ curl -u kibana:kibanapassword http://XXX.XX.XX.XX:9500/_xpack?pretty
{
"build" : {
"hash" : "dbf1a64",
"date" : "2017-09-07T03:27:09.260Z"
},
"license" : {
"uid" : "61885a70-5004-499e-b009-f8cd6f9b165d",
"type" : "trial",
"mode" : "trial",
"status" : "active",
"expiry_date_in_millis" : 1508515975830
},
"features" : {
"graph" : {
"description" : "Graph Data Exploration for the Elastic Stack",
"available" : true,
"enabled" : true
},
"ml" : {
"description" : "Machine Learning for the Elastic Stack",
"available" : true,
"enabled" : true,
"native_code_info" : {
"version" : "5.6.0",
"build_hash" : "93aea61f57f7d8"
}
},
"monitoring" : {
"description" : "Monitoring for the Elastic Stack",
"available" : true,
"enabled" : true
},
"security" : {
"description" : "Security for the Elastic Stack",
"available" : true,
"enabled" : true
},
"watcher" : {
"description" : "Alerting, Notification and Automation for the Elastic Stack",
"available" : true,
"enabled" : true
}
},
"tagline" : "You know, for X"
}

and yes, it stops it from happening. Anyway, in my case I'll need this monitoring tools.

this is the log I am retrieving from kibana
log [14:06:56.169] [warning][license][xpack] License information could not be obtained from Elasticsearch. [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate="Basic realm="security" charset="UTF-8"" } } :: {"path":"/_xpack","statusCode":401,"response":"{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_xpack]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_xpack]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}","wwwAuthenticateDirective":"Basic realm="security" charset="UTF-8""}

Am I missing some kind of configuration here?
I have defined user and password for kibana.


(André Ramos) #10

Fixed:

Added xpack.monitoring.elasticsearch.username and xpack.monitoring.elasticsearch.password to my kibana.yml, as described by @tsullivan in X-Pack Monitoring screen error after ES upgrade.

Thanks


(Tim Sullivan) #11

Thanks for the update, Andre!


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.