User in Kibana get 403

Hi Guys,

i create a role with the following permissions and assigned it to user :

  "webrtmon" : {
    "cluster" : [ ],
    "indices" : [
      {
        "names" : [
          "*"
        ],
        "privileges" : [
          "all",
          "manage",
          "monitor",
          "read",
          "index",
          "create",
          "delete",
          "write",
          "delete_index",
          "view_index_metadata",
          "create_index",
          "read_cross_cluster",
          "manage_follow_index"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [
      {
        "application" : "kibana-.kibana",
        "privileges" : [
          "all"
        ],
        "resources" : [
          "*"
        ]
      }
    ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

In Additional, i assigned to the user "kibana_user" role,
when i log in into kibana i got the following error:

<
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"can_match","grouped":true,"failed_shards":[{"shard":0,"index":"prod-eqx-webrtmon-2019.07.17","node":"tYmlAKs7Q-ivDnwmjnXOyA","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}},{"shard":0,"index":"prod-eqx-webrtmon-2019.07.18","node":"ikGwnTHESpK1cihK3QYXdA","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}},{"shard":0,"index":"prod-eqx-webrtmon-2019.07.19","node":"ikGwnTHESpK1cihK3QYXdA","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}},{"shard":0,"index":"prod-eqx-webrtmon-2019.07.20","node":"ikGwnTHESpK1cihK3QYXdA","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}},{"shard":0,"index":"prod-eqx-webrtmon-2019.07.21","node":"ikGwnTHESpK1cihK3QYXdA","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}},{"shard":0,"index":"prod-eqx-webrtmon-2019.07.22","node":"BpSucbW5Rf-fnByas2xNiA","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [webrtmon]"}},{"shard":0,"index":"prod-eqx-webrtmon-2019.07.23","node":"tYmlAKs7Q-ivDnwmjnXOyA","reason":{"type":"security_exception","reason":"action />

After i saw the following user i logged in with my superuser(elastic) and ran the command:

GET _XPACK/security/authenticate

once with my superuser and once with the user i created and i see the following diffs with my superuser the respond is
:

{
  "username" : "elastic",
  "roles" : [
    "superuser"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "lookup_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  }
}

And with my new user respond is :

{
  "username" : "webrtmon",
  "roles" : [
    "kibana_user",
    "webrtmon"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : { },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "default_native",
    "type" : "native"
  },
  "lookup_realm" : {
    "name" : "default_native",
    "type" : "native"
  }
}

i see they use different realm which i never configured, what could be the reason..?( it happens with every user i create)
and how can i fix it?

I can't reproduce this with what you gave us, by why do you use this long list of index privileges if you also add "all" to it? Using just "all" would suffice.

I tried with just all and still getting 403, the example I showed here is after many attempts
even when I gave just all still nothing happend ..

Apologies, but I can't replicate your issue. Can you start over with a new role definition ( a simple one ) and guide us through the steps that you take to get to this point where you get the errors ( and share the exact errors ) ?