SHIELD terms lookup filter : AuthorizationException BUG

Hi,

Using:

  • ElasticSearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the correct
security context.

This is very easy to reproduce, see gist :

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring to

the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4419d9d4-9bcc-4fab-afa3-a70799891f44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Bert,

Thank you for the detailed report and reproduction of this issue. This is a
known limitation with Shield and certain operations in elasticsearch. We're
working to resolve this in a future release.

We will be documenting this limitation and all of the operations affected
shortly; this was something that we had forgotten to document.

-Jay

On Monday, April 20, 2015 at 10:46:40 AM UTC-4, Bert Vermeiren wrote:

Hi,

Using:

  • Elasticsearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the correct
security context.

This is very easy to reproduce, see gist :

SHIELD terms lookup filter AuthorizationException BUG · GitHub

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring to

the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/670b111f-54d0-46e8-8b8e-6ec47e8d4a1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Jay,

Thanks to acknowledge !

Is there any way to work around this issue ? We definitely need a kind of
"join" filter for limiting the returned data based on some
permissions/tokens.

We are also starting discussions for a support and re-distribution license
with both your and our marketing organisation.

Is there any way to get a fix within a support contract ?

Thanks,

Regards, Bert.

Op woensdag 22 april 2015 14:34:07 UTC+2 schreef Jay Modi:

Hi Bert,

Thank you for the detailed report and reproduction of this issue. This is
a known limitation with Shield and certain operations in elasticsearch.
We're working to resolve this in a future release.

We will be documenting this limitation and all of the operations affected
shortly; this was something that we had forgotten to document.

-Jay

On Monday, April 20, 2015 at 10:46:40 AM UTC-4, Bert Vermeiren wrote:

Hi,

Using:

  • Elasticsearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the correct
security context.

This is very easy to reproduce, see gist :

SHIELD terms lookup filter AuthorizationException BUG · GitHub

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring to

the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4560604a-de24-4150-9c3a-c5386ed3087a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Bert,

I don't know of a workaround to accomplish this in a single query right
now. We have been discussing how to fix this issue in depth over the past
few days and have ideas on how to move forward but no timeline on it being
resolved.

Regarding support contracts and fixes, I'm going to defer that question to
the person your company is in contact with. They'll be able to answer that
much better than I can.

On Wednesday, April 22, 2015 at 9:15:21 AM UTC-4, Bert Vermeiren wrote:

Hi Jay,

Thanks to acknowledge !

Is there any way to work around this issue ? We definitely need a kind of
"join" filter for limiting the returned data based on some
permissions/tokens.

We are also starting discussions for a support and re-distribution license
with both your and our marketing organisation.

Is there any way to get a fix within a support contract ?

Thanks,

Regards, Bert.

Op woensdag 22 april 2015 14:34:07 UTC+2 schreef Jay Modi:

Hi Bert,

Thank you for the detailed report and reproduction of this issue. This is
a known limitation with Shield and certain operations in elasticsearch.
We're working to resolve this in a future release.

We will be documenting this limitation and all of the operations affected
shortly; this was something that we had forgotten to document.

-Jay

On Monday, April 20, 2015 at 10:46:40 AM UTC-4, Bert Vermeiren wrote:

Hi,

Using:

  • Elasticsearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the correct
security context.

This is very easy to reproduce, see gist :

SHIELD terms lookup filter AuthorizationException BUG · GitHub

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring

to the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/794f2fcf-6b47-41bd-878c-fbb048577c28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I'm having the same problem with Elasticsearch 1.4.5 with shield 1.1

On Thursday, April 23, 2015 at 2:03:23 PM UTC-5, Jay Modi wrote:

Hi Bert,

I don't know of a workaround to accomplish this in a single query right
now. We have been discussing how to fix this issue in depth over the past
few days and have ideas on how to move forward but no timeline on it being
resolved.

Regarding support contracts and fixes, I'm going to defer that question to
the person your company is in contact with. They'll be able to answer that
much better than I can.

On Wednesday, April 22, 2015 at 9:15:21 AM UTC-4, Bert Vermeiren wrote:

Hi Jay,

Thanks to acknowledge !

Is there any way to work around this issue ? We definitely need a kind of
"join" filter for limiting the returned data based on some
permissions/tokens.

We are also starting discussions for a support and re-distribution
license with both your and our marketing organisation.

Is there any way to get a fix within a support contract ?

Thanks,

Regards, Bert.

Op woensdag 22 april 2015 14:34:07 UTC+2 schreef Jay Modi:

Hi Bert,

Thank you for the detailed report and reproduction of this issue. This
is a known limitation with Shield and certain operations in elasticsearch.
We're working to resolve this in a future release.

We will be documenting this limitation and all of the operations
affected shortly; this was something that we had forgotten to document.

-Jay

On Monday, April 20, 2015 at 10:46:40 AM UTC-4, Bert Vermeiren wrote:

Hi,

Using:

  • Elasticsearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the
correct security context.

This is very easy to reproduce, see gist :

SHIELD terms lookup filter AuthorizationException BUG · GitHub

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring

to the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--

CONFIDENTIAL COMMUNICATION:

This email may contain confidential or legally privileged material, and is
for the sole use of the intended recipient. Use or distribution by an
unintended recipient is prohibited, and may be a violation of law. If you
believe that you received this email in error, please do not read, forward,
print or copy this email or any attachments. Please delete the email and
all attachments, and inform the sender that you have deleted the email and
all attachments. Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ca71964-2c03-4a26-8f1f-f63ac40269e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hello Jay,

I am having a similar issue using ES 1.4.3 + Shield 1.1. As soon as my
application instance (with embedded ES node.client) starts up, it throws :

Caused by: org.elasticsearch.shield.authz.AuthorizationException: action
[indices:admin/exists] is unauthorized for user [__es_system_user]

and therefore my application will not start. I have tried the following as
per the documentation here
(User Authentication | Shield [2.4] | Elastic)
:

shield.authc: anonymous: roles: admin

and

shield.authc: anonymous: username:
__es_system_user roles: admin

but they don't have any affect.

If there is any documentation I can read of the restriction(s) in
ES/Shield, please can you point me to it ? I need to understand if this is
something we can program/config around or whether we can patch Shield 1.1
(we only support v1.1 until next release of our application).

Thanks,

Steve.

On Wednesday, 22 April 2015 13:34:07 UTC+1, Jay Modi wrote:

Hi Bert,

Thank you for the detailed report and reproduction of this issue. This is
a known limitation with Shield and certain operations in elasticsearch.
We're working to resolve this in a future release.

We will be documenting this limitation and all of the operations affected
shortly; this was something that we had forgotten to document.

-Jay

On Monday, April 20, 2015 at 10:46:40 AM UTC-4, Bert Vermeiren wrote:

Hi,

Using:

  • Elasticsearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the correct
security context.

This is very easy to reproduce, see gist :

SHIELD terms lookup filter AuthorizationException BUG · GitHub

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring to

the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c5f81132-4cf7-4cfd-9f69-c3f47bb0b1d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

HI Steve,

I think we should move this to a new thread as it is probably a different
issue. May I suggest moving this to our new discussion forum with a section
specifically for Shield Elasticsearch - Discuss the Elastic Stack if you have any
followups to my response below?

To get you some help and on your way though, we do have documentation about
limitations [1] and node clients[2].

Also a few other notes, what you've defined for the anonymous user is
actually a different user than the actual system user, so it will not have
the affect you are looking for but instead will give every request without
any user full access. This may be a little confusing but we don't give our
internal system calls full rights, so what you're most probably doing in
your application is checking for the existence of an index somewhere
without specifying any credentials for the request. The details on
specifying credentials per request are located in [2] as well. The second
note, is have you considered switching to a transport client? With Shield
and node clients embedded in your application, you are essentially making
your application part of the cluster since it is a node and these nodes
need access to all of the Shield files such as users, roles, and the system
key.

[1] Appendix 3. Limitations | Shield - Elasticsearch Security Plugin [1.1] | Elastic
[2] Java clients | Shield - Elasticsearch Security Plugin [1.1] | Elastic

On Thursday, May 21, 2015 at 7:12:07 AM UTC-4, Steve Barnes wrote:

Hello Jay,

I am having a similar issue using ES 1.4.3 + Shield 1.1. As soon as my
application instance (with embedded ES node.client) starts up, it throws :

Caused by: org.elasticsearch.shield.authz.AuthorizationException: action
[indices:admin/exists] is unauthorized for user [__es_system_user]

and therefore my application will not start. I have tried the following as
per the documentation here (
User Authentication | Shield [2.4] | Elastic)
:

shield.authc: anonymous: roles: admin

and

shield.authc: anonymous: username:
__es_system_user roles: admin

but they don't have any affect.

If there is any documentation I can read of the restriction(s) in
ES/Shield, please can you point me to it ? I need to understand if this is
something we can program/config around or whether we can patch Shield 1.1
(we only support v1.1 until next release of our application).

Thanks,

Steve.

On Wednesday, 22 April 2015 13:34:07 UTC+1, Jay Modi wrote:

Hi Bert,

Thank you for the detailed report and reproduction of this issue. This is
a known limitation with Shield and certain operations in elasticsearch.
We're working to resolve this in a future release.

We will be documenting this limitation and all of the operations affected
shortly; this was something that we had forgotten to document.

-Jay

On Monday, April 20, 2015 at 10:46:40 AM UTC-4, Bert Vermeiren wrote:

Hi,

Using:

  • Elasticsearch 1.5.1
  • SHIELD 1.2

Whenever I use a terms lookup filter in a search query, I get an
UnAuthorizedException for the [__es_system_user] user although the actual
user has even 'admin' role privileges.
This seems a bug to me, where the terms filter does not have the correct
security context.

This is very easy to reproduce, see gist :

SHIELD terms lookup filter AuthorizationException BUG · GitHub

Scenario :

Add user 'admin' with default 'admin' role.

./bin/shield/esusers useradd admin -p admin1 -r admin

create index.

curl -XPUT 'admin:admin1@localhost:9200/customer'

create a document on the index

curl -XPUT 'admin:admin1@localhost:9200/customer/external/1' -d '
{
"name" : "John Doe",
"token" : "token1"
}'

create additional index for the "terms lookup" filter functionality

curl -XPUT 'admin:admin1@localhost:9200/tokens'

create document in 'tokens' index

curl -XPUT 'admin:admin1@localhost:9200/tokens/tokens/1' -d '
{
"group" : "1",
"tokens" : ["token1", "token2" ]
}'

search with a terms lookup filter on the "customer" index, referring

to the 'tokens' index.

curl -XGET 'admin:admin1@localhost:9200/customer/external/_search' -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"terms": {
"token": {
"index": "tokens",
"type": "tokens",
"id": "1",
"path": "tokens"
}
}
}
}
}
}'

=> org.elasticsearch.shield.authz.AuthorizationException: action
[indices:data/read/get] is unauthorized for user [__es_system_user]

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/408cb544-1f07-426a-8d23-f6308f33211d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.