API key and document-level security: Query not limiting response

Elastic Stack Elasticsearch
4

Hi @sgasse Couldn't help but poking in (interesting topic)...
and welcome to the community...

Document and Field level security is a platinum feature see here, but yes you should be OK to trial this.

So good news I just tested and for me it worked as expected in cloud with platinum license.

So I think we just need dig in a bit more.

Perhaps we should try a term query .... which can only match exact terms. its possible your match query is a full text query is is matching more than you think as from the Dev Tools it only returns the top 10 results.

  "query": {
    "term": {
      "project.keyword": {
        "value": "my_project"
      }
    }
  }

Here is my example
In my example I have many apps that are in a field cloudfoundry.app.name that field is a keyword if it was using the default mapping I would use cloudfoundry.app.name.keyword

POST /_security/api_key
{
  "name": "my-api-key",
  "expiration": "1d",
  "role_descriptors": {
    "role-a": {
      "cluster": [
        "all"
      ],
      "index": [
        {
          "names": [
            "filebeat-*"
          ],
          "privileges": [
            "read"
          ],
          "query": {
            "term": {
              "cloudfoundry.app.name": {
                "value": "spring-music"
              }
            }
          }
        }
      ]
    }
  }
}

Result

{
  "id" : "sadfasdfkBsdfsdfl0p6VzfPd",
  "name" : "my-api-key",
  "expiration" : 1620139846091,
  "api_key" : "asdadadsasdRde14awKNAgRZA"
}

converted to base64

echo -n "sadfasdfkBsdfsdfl0p6VzfPd:asdadadsasdRde14awKNAgRZA" | base64

then ran some queries

curl -H "Authorization: ApiKey sadfsdfsadfsadf2VnpmUGQ6Ukl6WTN6RmhSZGUxNGF3S05BZ1JaQQ==" -H "Content-Type: application/json" -d '{"size":3,"query":{"term":{"cloudfoundry.app.name":{"value":"spring-music"}}}}' https://myelasticurl.es.us-west1.gcp.cloud.es.io/filebeat-7.11.2-2021.04.28-000011/_search | jq

Worked as expected I got results for spring-music

Then I specifically tried to get results that we not matched the the role query i.e. search for other app specificallyy

sbrown$ curl -H "Authorization: ApiKey asdfsadfasdfasdfMHA2VnpmUGQ6Ukl6WTN6RmhSZGUxNGF3S05BZ1JaQQ==" -H "Content-Type: application/json" -d '{"size":3,"query":{"term":{"cloudfoundry.app.name":{"value":"scheduler-200ms"}}}}' https://myelasticurl.es.us-west1.gcp.cloud.es.io/filebeat-7.11.2-2021.04.28-000011/_search | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   242  100   161  100    81    540    271 --:--:-- --:--:-- --:--:--   812
{
  "took": 19,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}
ceres-2:filebeat-7.11.2-linux-x86_64 sbrown$

Then I tried a match_all and just got the results I would expect

ceres-2:filebeat-7.11.2-linux-x86_64 sbrown$ curl -H "Authorization: ApiKey sadfsadfsadfVnpmUGQ6Ukl6WTN6RmhSZGUxNGF3S05BZ1JaQQ==" -H "Content-Type: application/json" -d '{"size":3,"query":{"match_all":{}}}' https://myelasticurl.es.us-west1.gcp.cloud.es.io/filebeat-7.11.2-2021.04.28-000011/_search | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6256  100  6221  100    35  19440    109 --:--:-- --:--:-- --:--:-- 19489
{
  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "filebeat-7.11.2-2021.04.28-000011",
        "_type": "_doc",
        "_id": "MLbKGXkB_INU45ioej7I",
        "_score": 1,
        "_source": {
          "agent": {
            "hostname": "e8b823dc-4c45-4a00-7db7-95a4",
            "name": "e8b823dc-4c45-4a00-7db7-95a4",
            "id": "3e72667d-dbb0-46f1-a3cf-5460ee786dd9",
            "type": "filebeat",
            "ephemeral_id": "b3cf31bd-349c-42f6-b81c-b1ddaaab1cd9",
            "version": "7.11.2"
          },
          "message": "2021-04-28 18:42:49.910  INFO 20 --- [nio-8080-exec-1] o.c.samples.music.web.AlbumController    : Deleting album 7a41e2d3-689c-4f4f-a028-62ff5e30f10b",
          "input": {
            "type": "cloudfoundry"
          },
          "@timestamp": "2021-04-28T18:42:49.910Z",
          "ecs": {
            "version": "1.6.0"
          },
          "stream": "stdout",
          "host": {
            "name": "e8b823dc-4c45-4a00-7db7-95a4"
          },
          "cloudfoundry": {
            "app": {
              "name": "spring-music",
              "id": "ec3e9ff9-ff03-412e-98a9-40ed8d537241"
            },
...

Hope This Helps...