X-pack document level filtering not operating as expected

Base version information: this work is being done on an ELK stack running version 5.6.3 of elasticsearch, kibana, logstash, and x-pack.

I am trying to define multiple levels of roles that can be combined to allow users to exist at 4 levels:

  1. admin_full_access - this kibana user can manage all indices. No document level filtering is required on any index.
  2. admin_limited_access - this kibana user can manage all indices, except in one index there is a field that they are not allowed to view documents if that field value matches a specific value. (So document level filtering is needed for just that single index. Other indices do not contain the field in question).
  3. user_full_access - this kibana user can read a subset of indices. No document level filtering is required - they can see all documents of the subset of indices.
  4. user_limited_access - this kibana user can read the same subset of indices as the user_full_access can. Document level filtering the same as with the admin_limited_access user is required for the same specific index.

I am using the file based realm to define the roles for these users because I do not want anyone modifying them from the kibana dashboard. Only a user who can actually ssh into the elasticsearch instances should be able to make those modifications.

Here is essentially what I have set up for the definitions of the roles/users:

base_admin_role:
    cluster: ['all']
    indices:
      - names: ['index-that-is-limited*']
        privileges: ['all']
        query: '{"match":{"field-name":"field-value"}}'

base_user_role:
    cluster: ['manage']
    indices:
      - names: ['index-that-is-limited*']
        privileges: ['read']
        query: '{"match":{"field-name":"field-value"}}'

full_admin_index_role:
    indices:
      - names: ['*']
        privileges: ['all']

full_user_index_role:
    indices:
      - names: ['index-that-is-limited', 'the-remaining', 'subset-of', 'indices-that', 'I-want', 'to-have', 'access-to']
        privileges: ['read']

limited_admin_index_role:
    indices:
      - names: ['all-other-indices']
        privileges: ['all']

limited_user_index_role:
    indices:
      - names: ['the-remaining', 'subset-of', 'indices-that', 'I-want', 'to-have', 'access-to']
        privileges: ['read']

And the users are then given roles accordingly:
admin_full_access: kibana_dashboard, base_admin_role, full_admin_index_role
admin_limited_access: kibana_dashboard, base_admin_role, limited_admin_index_role
user_full_access: kibana_dashboard, base_user_role, full_user_index_role
user_limited_access: kibana_dashboard, base_user_role, limited_user_index_role

Which seems like a logical way to construct the permissions. However, when I log into kibana using any of the four roles above, they all have full access to every index and document out there. I thought maybe it was the kibana_user role causing the problem so removed it and then queried elasticsearch directly using the users/passwords for each to see if the index/document limitations were working that way but they were not. All four users were still able to access all indices and documents regardless of their user or the content of the field that the queries are set up on.

Please help. I am thoroughly stumped at this point.

Can you updatte your post to wrap your role defintions in a code block? (The </> button).
It's hard to be sure if your config is correct without the code formatting.

This is not the cause of your issue, but is a problem with your setup:

indices:
   - names: ['index-that-is-limited*']
     privileges: ['all']
     query: '{"match":{"field-name":"field-value"}}'

You cannot combine write privileges with document level security. You really need to set the privileges on this role to just read

Updated. My browser (on a work computer, so I am not allowed to update it) does not show the options when editing properly. They are all just blank spots that you have to hover over to see the tool tip in order to know what they are for!
I will update the privileges to be read only and see if that makes a difference.
Thank you for your assistance, Tim!

I updated the privileges for the index-that-is-limited to be 'read' but still have the same problem happening. I have refactored the roles several times, even simplifying down to a single index that is accessible with the same query as above yet when I log in as either of the users (user_full_access or user_limited_access) in kibana I am still able to see every document. Even those that I would expect to be filtered.

I'm not sure why this isn't working for you.
I replicated your setup and everything behaves as expected.

What exactly are you doing & seeing in Kibana?
Can you try and verify your setup works like my example below using the Kibana console (dev tools) or curl? That will help work out whether you have a role setup problem, or whether something in Kibana isn't working the way you expect it to.

roles.yml

base_admin_role:
    cluster: ['all']
    indices:
      - names: ['limited-*']
        privileges: ['read']
        query: '{"match":{"visible":"yes"}}'

base_user_role:
    cluster: ['manage']
    indices:
      - names: ['limited-*']
        privileges: ['read']
        query: '{"match":{"visible":"yes"}}'

full_admin_index_role:
    indices:
      - names: ['*']
        privileges: ['all']

full_user_index_role:
    indices:
      - names: ['limited-*', 'other-*' ]
        privileges: ['read']

limited_admin_index_role:
    indices:
      - names: ['other-*']
        privileges: ['all']

limited_user_index_role:
    indices:
      - names: ['other-*']
        privileges: ['read']

Users

$ bin/x-pack/users list
user1          : full_user_index_role,base_user_role
user2          : limited_user_index_role,base_user_role
admin1         : full_admin_index_role,base_admin_role
admin2         : limited_admin_index_role,base_admin_role

Documents:

curl -uelastic 'http://localhost:9200/limited-1/doc/_search?pretty'
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "limited-1",
        "_type" : "doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "name" : "visible-doc-1",
          "visible" : "yes"
        }
      },
      {
        "_index" : "limited-1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "name" : "hidden-doc-1",
          "visible" : "no"
        }
      }
    ]
  }
}

As full admin

curl -uadmin1 'http://localhost:9200/limited-1/doc/_search'
{"took":1,"timed_out":false,
  "_shards":{"total":5,"successful":5,"skipped":0,"failed":0},
  "hits":{"total":2,"max_score":1.0,
    "hits":[
      {"_index":"limited-1","_type":"doc","_id":"2","_score":1.0,"_source":{ "name" : "visible-doc-1", "visible" : "yes" }},
      {"_index":"limited-1","_type":"doc","_id":"1","_score":1.0,"_source":{ "name" : "hidden-doc-1", "visible" : "no" }}
]}}

As limited admin

curl -uadmin2 'http://localhost:9200/limited-1/doc/_search'
{"took":1,"timed_out":false,
  "_shards":{"total":5,"successful":5,"skipped":0,"failed":0},
  "hits":{"total":1,"max_score":1.0,
    "hits":[
      {"_index":"limited-1","_type":"doc","_id":"2","_score":1.0,"_source":{ "name" : "visible-doc-1", "visible" : "yes" }}
]}}

As full user

curl -uuser1 'http://localhost:9200/limited-1/doc/_search'
{"took":1,"timed_out":false,
  "_shards":{"total":5,"successful":5,"skipped":0,"failed":0},
  "hits":{"total":2,"max_score":1.0,
    "hits":[
      {"_index":"limited-1","_type":"doc","_id":"2","_score":1.0,"_source":{ "name" : "visible-doc-1", "visible" : "yes" }},
      {"_index":"limited-1","_type":"doc","_id":"1","_score":1.0,"_source":{ "name" : "hidden-doc-1", "visible" : "no" }}
]}}

As limited user

curl -uuser2 'http://localhost:9200/limited-1/doc/_search'
{"took":1,"timed_out":false,
  "_shards":{"total":5,"successful":5,"skipped":0,"failed":0},
  "hits":{"total":1,"max_score":1.0,
    "hits":[
      {"_index":"limited-1","_type":"doc","_id":"2","_score":1.0,"_source":{ "name" : "visible-doc-1", "visible" : "yes" }}
]}}

Hi Tim -

What I am seeing (after playing around with the queries a bit more but not changing the configuration from that shown above for both of us) is that the limits on which indices my user_* accounts can access are correct (meaning that if I do a search on an index I don't have authorization for then I get nothing back in my hits), but that the query constraint that is supposed to limit my *_limited accounts into getting a smaller subset of documents in return is not working as expected. I get the full set of documents back (so in your example, I would get both documents back). I have confirmed that I do not have a typo in the field name and that the value of the field is what is expected (in your example above, the field name is correctly listed as "visible" and the value is "yes") but I still get the documents with the value as "no" in my hits.

Actually, I take that back. Even for the index constraints I am still receiving documents back when I have not authorized the user to access the index.

I am seeing the same behavior as far as what the users can see whether I do a curl command or log in to the kibana dashboard and look at things there.
For example, if I query as admin2 from your example I get the same response as if I queried as admin1
Similarly, if I log in to the kibana dashboard as admin2, I can still see the "visible": "no" record in my Discover tab even though I would expect to see only the "visible": "yes" record there.

What sort of license are you using?

GET /_xpack/license and look at the "type" field.

Platinum.

So it should not be a problem with a lack of permission to use the Security features.

The license is valid through April 2019 as well.

At this point I think it might be best if you raise the issue through our support team.

The DLS feature works, but there is something particular about your setup that is causing a problem.
It's not really possible to debug it any further without seeing exactly what your configuration looks like, and we'd prefer to do that on a private channel, with someone who has the time to dedicate to it.

You can refer your support engineer to this thread as a starting point (and they will know how to contact me if they need more info).

How do I contact the support team?

I appreciate your initial efforts in this matter - you were definitely helpful with confirming that I am not way off base in the way I am defining the roles and users, which is valuable by itself.

Thanks.

The support portal link is available at the bottom of our website https://www.elastic.co. You will need credentials to login; if you do not have these credentials, I suggest getting in touch with the person that provided you with the license to get added as a support contact.

Ok, I will contact our licensing person to see about getting credentials. Thank you!

Figured I would update this thread with the solution that the support team and I discovered.

If you have the anonymous user enabled (which we do) then all other users in the system get the anonymous user's permissions by default. In our system, we had anonymous set to superuser due to some program complaints and issues when we had locked it down earlier. So that meant that all of our defined users (admin1, admin2, user1, user2, etc) all by default also had superuser authority.

So just something for everyone to keep in mind - anonymous user permissions are automatically attached to all other users by x-pack, and there is no way to change this behavior.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.