Fine-grained security for ES-based API


(Shane Witbeck) #1

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed documents
    which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource seem
to be candidates but it's unclear what exactly they do. Could you explain
them?

Thanks!


(mat) #2

Not a direct answer to your question but I am using a simple abstraction to
implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.

The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.

For example a policy query could then only be registered

  • allow all requests from a specific IP address
  • Use script filters to allow all Update requests only for documents with
    an 'owner' that match user Id supplied as a parameter
  • all access to docs for a limited time
  • only allow access to docs when specific cookies or tokens are available
    in the header or session.
  • only allow queries which request specific fields
  • only updates to specific fields of docs which a user has read access to.

etc..

If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.

Thanks,
mat

On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!


(Shay Banon) #3

The way search request works is that you can setSource and setExtraSource,
first source is parsed, and then the extra source is parsed overriding or
adding specific request parameters. For example, you can have the query set
in the source json, and the fields set in the extra source one. Note, that
if, for example, query is set on both, then extra source will override the
one in source.

On Thu, May 24, 2012 at 5:56 PM, Shane Witbeck shane@digitalsanctum.comwrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!


(Shane Witbeck) #4

Thanks for explaining source vs. extraSource. Is there a way to parse the
source (which would be the query) ? I'd like to be able to determine if
things like the fields SHOULD be overwritten. For example, if the user
specified fields then I will opt to not override them. I opened an issue
about this the other day here:

Thanks.

On Tuesday, May 29, 2012 2:43:34 PM UTC-4, kimchy wrote:

The way search request works is that you can setSource and setExtraSource,
first source is parsed, and then the extra source is parsed overriding or
adding specific request parameters. For example, you can have the query set
in the source json, and the fields set in the extra source one. Note, that
if, for example, query is set on both, then extra source will override the
one in source.

On Thu, May 24, 2012 at 5:56 PM, Shane Witbeck shane@digitalsanctum.comwrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder
via the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!


(Shane Witbeck) #5

Mat,

Thanks for your reply and outlining your approach. Although your approach
seems effective I was looking for a more simplified approach using just the
ES API.

Shane

On Sunday, May 27, 2012 12:20:02 PM UTC-4, mat taylor wrote:

Not a direct answer to your question but I am using a simple abstraction
to implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.

The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.

For example a policy query could then only be registered

  • allow all requests from a specific IP address
  • Use script filters to allow all Update requests only for documents with
    an 'owner' that match user Id supplied as a parameter
  • all access to docs for a limited time
  • only allow access to docs when specific cookies or tokens are available
    in the header or session.
  • only allow queries which request specific fields
  • only updates to specific fields of docs which a user has read access to.

etc..

If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.

Thanks,
mat

On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!


(Shane Witbeck) #6

For my use case, the query and fields might be part of the same request so
setting the query in the source json and fields set in the extra source may
not work. It seems like there should be a parse method in the ES code to
parse JSON requests. This would allow me to easily determine if the user
specified fields.

On Tuesday, May 29, 2012 2:43:34 PM UTC-4, kimchy wrote:

The way search request works is that you can setSource and setExtraSource,
first source is parsed, and then the extra source is parsed overriding or
adding specific request parameters. For example, you can have the query set
in the source json, and the fields set in the extra source one. Note, that
if, for example, query is set on both, then extra source will override the
one in source.

On Thu, May 24, 2012 at 5:56 PM, Shane Witbeck shane@digitalsanctum.comwrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder
via the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!


(Mil Werns) #7

Hi Mat,

Your approach sounds very interesting. Could you please give me some
example code (e.g. for the "owner" and "specific fields" types)?

Thank you
Mil

On Sunday, May 27, 2012 6:20:02 PM UTC+2, mat taylor wrote:

Not a direct answer to your question but I am using a simple abstraction
to implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.

The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.

For example a policy query could then only be registered

  • allow all requests from a specific IP address
  • Use script filters to allow all Update requests only for documents with
    an 'owner' that match user Id supplied as a parameter
  • all access to docs for a limited time
  • only allow access to docs when specific cookies or tokens are available
    in the header or session.
  • only allow queries which request specific fields
  • only updates to specific fields of docs which a user has read access to.

etc..

If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.

Thanks,
mat

On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!

--


(Hendrik) #8

Maybe this is interesting
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/tavroa3Nw5g

Am Freitag, 12. Oktober 2012 22:34:53 UTC+2 schrieb S. Milwerns:

Hi Mat,

Your approach sounds very interesting. Could you please give me some
example code (e.g. for the "owner" and "specific fields" types)?

Thank you
Mil

On Sunday, May 27, 2012 6:20:02 PM UTC+2, mat taylor wrote:

Not a direct answer to your question but I am using a simple abstraction
to implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.

The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.

For example a policy query could then only be registered

  • allow all requests from a specific IP address
  • Use script filters to allow all Update requests only for documents with
    an 'owner' that match user Id supplied as a parameter
  • all access to docs for a limited time
  • only allow access to docs when specific cookies or tokens are available
    in the header or session.
  • only allow queries which request specific fields
  • only updates to specific fields of docs which a user has read access to.

etc..

If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.

Thanks,
mat

On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:

I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:

  1. allow users to query ES via search but only return indexed
    documents which they have permission to read
  2. filter sensitive fields from the results. I'm doing this via
    something like: searchRequestBuilder.addPartialField("apiFields", null,
    FIELDS_TO_EXCLUDE);

For the implementation I'm basically composing a SearchRequestBuilder
via the Java API.

Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?

SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #9