Security: Authentication failed on tribe node

Hi,

We are on alpha5, and not able to authenticate when trying to perform any command from the tribe node, for eg.

# curl -u elastic:elastic -XGET localhost:9200

It would throw the following response:

{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}[

The same exact command runs just fine when executed on any of the data/master nodes:

{
  "name" : "dcmarhel6a",
  "cluster_name" : "epicon_elasticsearch_dc",
  "version" : {
    "number" : "5.0.0-alpha5",
    "build_hash" : "d327dd4",
    "build_date" : "2016-08-04T08:59:39.568Z",
    "build_snapshot" : false,
    "lucene_version" : "6.1.0"
  },
  "tagline" : "You Know, for Search"
}

We have 3 data nodes and 1 client (tribe) node and the cluster connection is working just fine as shown on the output below:

{
  "cluster_name" : "epicon_elasticsearch_dc",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 4,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

We're using native realm for the authentication, and have disabled the other realms. We've also made sure all elasticsearch.yml configuration are the same across the cluster, except for the tribe node:

node.data: false
node.master: false
node.name: ${HOSTNAME}
transport.tcp.port: 9300
http.port: 9200
bootstrap.memory_lock: true
discovery.zen.minimum_master_nodes: 2

xpack.security.audit.enabled: true

xpack.security.authc.realms.file.type: file
xpack.security.authc.realms.file.order: 0
xpack.security.authc.realms.file.enabled: false

xpack.security.authc.realms.native1.type: native
xpack.security.authc.realms.native1.order: 1
xpack.security.authc.realms.native1.enabled: true

tribe.dc.cluster.name: epicon_elasticsearch_dc
tribe.dc.network.host: 192.167.168.95
tribe.dc.discovery.zen.ping.unicast.hosts: ["192.167.168.91", "192.167.168.92", "192.167.168.93", "192.167.168.95"]

Wonder if we're missing something here..

There is a bug that prevents native and built in users from working on tribe nodes right now. As a workaround, you can fallback to the file based users until it is resolved.

Thanks for checking, we resorted to use the ldap realm instead. It's working just fine so far.