Security: Authentication failed on tribe node


We are on alpha5, and not able to authenticate when trying to perform any command from the tribe node, for eg.

# curl -u elastic:elastic -XGET localhost:9200

It would throw the following response:

{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}[

The same exact command runs just fine when executed on any of the data/master nodes:

  "name" : "dcmarhel6a",
  "cluster_name" : "epicon_elasticsearch_dc",
  "version" : {
    "number" : "5.0.0-alpha5",
    "build_hash" : "d327dd4",
    "build_date" : "2016-08-04T08:59:39.568Z",
    "build_snapshot" : false,
    "lucene_version" : "6.1.0"
  "tagline" : "You Know, for Search"

We have 3 data nodes and 1 client (tribe) node and the cluster connection is working just fine as shown on the output below:

  "cluster_name" : "epicon_elasticsearch_dc",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 4,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

We're using native realm for the authentication, and have disabled the other realms. We've also made sure all elasticsearch.yml configuration are the same across the cluster, except for the tribe node: false
node.master: false ${HOSTNAME}
transport.tcp.port: 9300
http.port: 9200
bootstrap.memory_lock: true
discovery.zen.minimum_master_nodes: 2 true file 0 false native 1 true epicon_elasticsearch_dc ["", "", "", ""]

Wonder if we're missing something here..

There is a bug that prevents native and built in users from working on tribe nodes right now. As a workaround, you can fallback to the file based users until it is resolved.

Thanks for checking, we resorted to use the ldap realm instead. It's working just fine so far.