I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed documents
which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource seem
to be candidates but it's unclear what exactly they do. Could you explain
them?
Not a direct answer to your question but I am using a simple abstraction to
implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.
The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.
For example a policy query could then only be registered
allow all requests from a specific IP address
Use script filters to allow all Update requests only for documents with
an 'owner' that match user Id supplied as a parameter
all access to docs for a limited time
only allow access to docs when specific cookies or tokens are available
in the header or session.
only allow queries which request specific fields
only updates to specific fields of docs which a user has read access to.
etc..
If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.
Thanks,
mat
On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
The way search request works is that you can setSource and setExtraSource,
first source is parsed, and then the extra source is parsed overriding or
adding specific request parameters. For example, you can have the query set
in the source json, and the fields set in the extra source one. Note, that
if, for example, query is set on both, then extra source will override the
one in source.
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
Thanks for explaining source vs. extraSource. Is there a way to parse the
source (which would be the query) ? I'd like to be able to determine if
things like the fields SHOULD be overwritten. For example, if the user
specified fields then I will opt to not override them. I opened an issue
about this the other day here:
Thanks.
On Tuesday, May 29, 2012 2:43:34 PM UTC-4, kimchy wrote:
The way search request works is that you can setSource and setExtraSource,
first source is parsed, and then the extra source is parsed overriding or
adding specific request parameters. For example, you can have the query set
in the source json, and the fields set in the extra source one. Note, that
if, for example, query is set on both, then extra source will override the
one in source.
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder
via the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
Thanks for your reply and outlining your approach. Although your approach
seems effective I was looking for a more simplified approach using just the
ES API.
Shane
On Sunday, May 27, 2012 12:20:02 PM UTC-4, mat taylor wrote:
Not a direct answer to your question but I am using a simple abstraction
to implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.
The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.
For example a policy query could then only be registered
allow all requests from a specific IP address
Use script filters to allow all Update requests only for documents with
an 'owner' that match user Id supplied as a parameter
all access to docs for a limited time
only allow access to docs when specific cookies or tokens are available
in the header or session.
only allow queries which request specific fields
only updates to specific fields of docs which a user has read access to.
etc..
If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.
Thanks,
mat
On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
For my use case, the query and fields might be part of the same request so
setting the query in the source json and fields set in the extra source may
not work. It seems like there should be a parse method in the ES code to
parse JSON requests. This would allow me to easily determine if the user
specified fields.
On Tuesday, May 29, 2012 2:43:34 PM UTC-4, kimchy wrote:
The way search request works is that you can setSource and setExtraSource,
first source is parsed, and then the extra source is parsed overriding or
adding specific request parameters. For example, you can have the query set
in the source json, and the fields set in the extra source one. Note, that
if, for example, query is set on both, then extra source will override the
one in source.
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder
via the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
Your approach sounds very interesting. Could you please give me some
example code (e.g. for the "owner" and "specific fields" types)?
Thank you
Mil
On Sunday, May 27, 2012 6:20:02 PM UTC+2, mat taylor wrote:
Not a direct answer to your question but I am using a simple abstraction
to implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.
The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.
For example a policy query could then only be registered
allow all requests from a specific IP address
Use script filters to allow all Update requests only for documents with
an 'owner' that match user Id supplied as a parameter
all access to docs for a limited time
only allow access to docs when specific cookies or tokens are available
in the header or session.
only allow queries which request specific fields
only updates to specific fields of docs which a user has read access to.
etc..
If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.
Thanks,
mat
On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder via
the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
Am Freitag, 12. Oktober 2012 22:34:53 UTC+2 schrieb S. Milwerns:
Hi Mat,
Your approach sounds very interesting. Could you please give me some
example code (e.g. for the "owner" and "specific fields" types)?
Thank you
Mil
On Sunday, May 27, 2012 6:20:02 PM UTC+2, mat taylor wrote:
Not a direct answer to your question but I am using a simple abstraction
to implement a fast, very flexible and fine grained security over elastic
search using a percolater proxy.
The idea is to use a proxy (I am usng nginx + lua) to represent each http
request as a json document with top level fields such as 'headers', 'body',
'path', 'session', and 'method', 'time' and then request authorisation for
this request by percolating this document through an index of 'policy'
filters which may or may not match.
For example a policy query could then only be registered
allow all requests from a specific IP address
Use script filters to allow all Update requests only for documents with
an 'owner' that match user Id supplied as a parameter
all access to docs for a limited time
only allow access to docs when specific cookies or tokens are available
in the header or session.
only allow queries which request specific fields
only updates to specific fields of docs which a user has read access to.
etc..
If no direct authorization for a request is found, I then the proxy will
fallback to a set of rewrite rules to add/remove query, field and
partial_field parameters to the request to on a per index/type basis. This
allows sensitive fields to be removed from any search results as well as
restricting searches as appropriate.
Thanks,
mat
On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote:
I'm building a minimal abstraction around ES to accomplish fine-grained
security control around searches based on indexed fields. The requirements
are:
allow users to query ES via search but only return indexed
documents which they have permission to read
filter sensitive fields from the results. I'm doing this via
something like: searchRequestBuilder.addPartialField("apiFields", null,
FIELDS_TO_EXCLUDE);
For the implementation I'm basically composing a SearchRequestBuilder
via the Java API.
Given a search query in JSON format, what's the best method to set the
query on the SearchRequestBuilder object while still being able to add my
filter(s) and exclude fields from the search results?
SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource
seem to be candidates but it's unclear what exactly they do. Could you
explain them?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.