# Fine-grained security for ES-based API **URL:** \ **Category:** Elasticsearch\ **Created:** [May 24, 2012, 3:56pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844 "2012-05-24T15:56:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [May 24, 2012, 3:56pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/1 "2012-05-24T15:56:59Z")
I'm building a minimal abstraction around ES to accomplish fine-grained security control around searches based on indexed fields. The requirements are: 1. allow users to query ES via search but only return indexed documents which they have permission to read 2. filter sensitive fields from the results. I'm doing this via something like: searchRequestBuilder.addPartialField("apiFields", null, FIELDS\_TO\_EXCLUDE); For the implementation I'm basically composing a SearchRequestBuilder via the Java API. Given a search query in JSON format, what's the best method to set the query on the SearchRequestBuilder object while still being able to add my filter(s) and exclude fields from the search results? SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource seem to be candidates but it's unclear what exactly they do. Could you explain them? Thanks! ---
**Author:** ![mat1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mat1/32/1679_2.png) [@mat1](https://discuss.elastic.co/u/mat1)\ **Post date:** [May 27, 2012, 4:20pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/2 "2012-05-27T16:20:02Z")
Not a direct answer to your question but I am using a simple abstraction to implement a fast, very flexible and fine grained security over elastic search using a percolater proxy. The idea is to use a proxy (I am usng nginx + lua) to represent each http request as a json document with top level fields such as 'headers', 'body', 'path', 'session', and 'method', 'time' and then request authorisation for this request by percolating this document through an index of 'policy' filters which may or may not match. For example a policy query could then only be registered - allow all requests from a specific IP address - Use script filters to allow all Update requests only for documents with an 'owner' that match user Id supplied as a parameter - all access to docs for a limited time - only allow access to docs when specific cookies or tokens are available in the header or session. - only allow queries which request specific fields - only updates to specific fields of docs which a user has read access to. etc.. If no direct authorization for a request is found, I then the proxy will fallback to a set of rewrite rules to add/remove query, field and partial\_field parameters to the request to on a per index/type basis. This allows sensitive fields to be removed from any search results as well as restricting searches as appropriate. Thanks, mat On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote: > I'm building a minimal abstraction around ES to accomplish fine-grained > security control around searches based on indexed fields. The requirements > are: > > 1. allow users to query ES via search but only return indexed > documents which they have permission to read > 2. filter sensitive fields from the results. I'm doing this via > something like: searchRequestBuilder.addPartialField("apiFields", null, > FIELDS\_TO\_EXCLUDE); > > For the implementation I'm basically composing a SearchRequestBuilder via > the Java API. > > Given a search query in JSON format, what's the best method to set the > query on the SearchRequestBuilder object while still being able to add my > filter(s) and exclude fields from the search results? > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > seem to be candidates but it's unclear what exactly they do. Could you > explain them? > > Thanks! ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 29, 2012, 6:43pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/3 "2012-05-29T18:43:34Z")
The way search request works is that you can setSource and setExtraSource, first source is parsed, and then the extra source is parsed overriding or adding specific request parameters. For example, you can have the query set in the source json, and the fields set in the extra source one. Note, that if, for example, query is set on both, then extra source will override the one in source. On Thu, May 24, 2012 at 5:56 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > I'm building a minimal abstraction around ES to accomplish fine-grained > security control around searches based on indexed fields. The requirements > are: > > 1. allow users to query ES via search but only return indexed > documents which they have permission to read > 2. filter sensitive fields from the results. I'm doing this via > something like: searchRequestBuilder.addPartialField("apiFields", null, > FIELDS\_TO\_EXCLUDE); > > For the implementation I'm basically composing a SearchRequestBuilder via > the Java API. > > Given a search query in JSON format, what's the best method to set the > query on the SearchRequestBuilder object while still being able to add my > filter(s) and exclude fields from the search results? > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > seem to be candidates but it's unclear what exactly they do. Could you > explain them? > > Thanks! ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [May 29, 2012, 7:28pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/4 "2012-05-29T19:28:55Z")
Thanks for explaining source vs. extraSource. Is there a way to parse the source (which would be the query) ? I'd like to be able to determine if things like the fields SHOULD be overwritten. For example, if the user specified fields then I will opt to not override them. I opened an issue about this the other day here: > > > I posted the following on the mailing list: https://groups.google.com/forum/?fro…mgroups#!topic/elasticsearch/lAmxGvP3pos > > I'd like to suggest an implementation which allows override "fields" or "partial\_fields" in order to accomplish filtering of certain fields (which are used for fine-grained security). > > For example: > 1. if a user passes in a query which has no "fields" or "partial\_fields" defined, use a predefined "partial\_fields" to prevent sensitive fields from appearing in search results. > 2. if a user passes a query which has "fields" or "partial\_fields", then merge or override fields available in search results. > > In other posts I have seen the suggestion to use indexed fields to accomplish fine-grained security. This works well except for the above scenario for trying to filter security-centric fields from search results. > > I look forward to hearing about an existing feature or new feature to accomplish the above. Thanks. On Tuesday, May 29, 2012 2:43:34 PM UTC-4, kimchy wrote: > The way search request works is that you can setSource and setExtraSource, > first source is parsed, and then the extra source is parsed overriding or > adding specific request parameters. For example, you can have the query set > in the source json, and the fields set in the extra source one. Note, that > if, for example, query is set on both, then extra source will override the > one in source. > > On Thu, May 24, 2012 at 5:56 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > I'm building a minimal abstraction around ES to accomplish fine-grained > > security control around searches based on indexed fields. The requirements > > are: > > > > 1. allow users to query ES via search but only return indexed > > documents which they have permission to read > > 2. filter sensitive fields from the results. I'm doing this via > > something like: searchRequestBuilder.addPartialField("apiFields", null, > > FIELDS\_TO\_EXCLUDE); > > > > For the implementation I'm basically composing a SearchRequestBuilder > > via the Java API. > > > > Given a search query in JSON format, what's the best method to set the > > query on the SearchRequestBuilder object while still being able to add my > > filter(s) and exclude fields from the search results? > > > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > > seem to be candidates but it's unclear what exactly they do. Could you > > explain them? > > > > Thanks! ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [May 29, 2012, 7:31pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/5 "2012-05-29T19:31:30Z")
Mat, Thanks for your reply and outlining your approach. Although your approach seems effective I was looking for a more simplified approach using just the ES API. Shane On Sunday, May 27, 2012 12:20:02 PM UTC-4, mat taylor wrote: > Not a direct answer to your question but I am using a simple abstraction > to implement a fast, very flexible and fine grained security over elastic > search using a percolater proxy. > > The idea is to use a proxy (I am usng nginx + lua) to represent each http > request as a json document with top level fields such as 'headers', 'body', > 'path', 'session', and 'method', 'time' and then request authorisation for > this request by percolating this document through an index of 'policy' > filters which may or may not match. > > For example a policy query could then only be registered > > - allow all requests from a specific IP address > - Use script filters to allow all Update requests only for documents with > an 'owner' that match user Id supplied as a parameter > - all access to docs for a limited time > - only allow access to docs when specific cookies or tokens are available > in the header or session. > - only allow queries which request specific fields > - only updates to specific fields of docs which a user has read access to. > > etc.. > > If no direct authorization for a request is found, I then the proxy will > fallback to a set of rewrite rules to add/remove query, field and > partial\_field parameters to the request to on a per index/type basis. This > allows sensitive fields to be removed from any search results as well as > restricting searches as appropriate. > > Thanks, > mat > > On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote: > > > I'm building a minimal abstraction around ES to accomplish fine-grained > > security control around searches based on indexed fields. The requirements > > are: > > > > 1. allow users to query ES via search but only return indexed > > documents which they have permission to read > > 2. filter sensitive fields from the results. I'm doing this via > > something like: searchRequestBuilder.addPartialField("apiFields", null, > > FIELDS\_TO\_EXCLUDE); > > > > For the implementation I'm basically composing a SearchRequestBuilder via > > the Java API. > > > > Given a search query in JSON format, what's the best method to set the > > query on the SearchRequestBuilder object while still being able to add my > > filter(s) and exclude fields from the search results? > > > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > > seem to be candidates but it's unclear what exactly they do. Could you > > explain them? > > > > Thanks! ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [June 6, 2012, 7:23pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/6 "2012-06-06T19:23:47Z")
For my use case, the query and fields might be part of the same request so setting the query in the source json and fields set in the extra source may not work. It seems like there should be a parse method in the ES code to parse JSON requests. This would allow me to easily determine if the user specified fields. On Tuesday, May 29, 2012 2:43:34 PM UTC-4, kimchy wrote: > The way search request works is that you can setSource and setExtraSource, > first source is parsed, and then the extra source is parsed overriding or > adding specific request parameters. For example, you can have the query set > in the source json, and the fields set in the extra source one. Note, that > if, for example, query is set on both, then extra source will override the > one in source. > > On Thu, May 24, 2012 at 5:56 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > I'm building a minimal abstraction around ES to accomplish fine-grained > > security control around searches based on indexed fields. The requirements > > are: > > > > 1. allow users to query ES via search but only return indexed > > documents which they have permission to read > > 2. filter sensitive fields from the results. I'm doing this via > > something like: searchRequestBuilder.addPartialField("apiFields", null, > > FIELDS\_TO\_EXCLUDE); > > > > For the implementation I'm basically composing a SearchRequestBuilder > > via the Java API. > > > > Given a search query in JSON format, what's the best method to set the > > query on the SearchRequestBuilder object while still being able to add my > > filter(s) and exclude fields from the search results? > > > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > > seem to be candidates but it's unclear what exactly they do. Could you > > explain them? > > > > Thanks! ---
**Author:** ![Mil\_Werns](https://avatars.discourse-cdn.com/v4/letter/m/aca169/32.png) [@Mil\_Werns](https://discuss.elastic.co/u/Mil_Werns)\ **Post date:** [October 12, 2012, 8:34pm UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/7 "2012-10-12T20:34:53Z")
Hi Mat, Your approach sounds very interesting. Could you please give me some example code (e.g. for the "owner" and "specific fields" types)? Thank you Mil On Sunday, May 27, 2012 6:20:02 PM UTC+2, mat taylor wrote: > Not a direct answer to your question but I am using a simple abstraction > to implement a fast, very flexible and fine grained security over elastic > search using a percolater proxy. > > The idea is to use a proxy (I am usng nginx + lua) to represent each http > request as a json document with top level fields such as 'headers', 'body', > 'path', 'session', and 'method', 'time' and then request authorisation for > this request by percolating this document through an index of 'policy' > filters which may or may not match. > > For example a policy query could then only be registered > > - allow all requests from a specific IP address > - Use script filters to allow all Update requests only for documents with > an 'owner' that match user Id supplied as a parameter > - all access to docs for a limited time > - only allow access to docs when specific cookies or tokens are available > in the header or session. > - only allow queries which request specific fields > - only updates to specific fields of docs which a user has read access to. > > etc.. > > If no direct authorization for a request is found, I then the proxy will > fallback to a set of rewrite rules to add/remove query, field and > partial\_field parameters to the request to on a per index/type basis. This > allows sensitive fields to be removed from any search results as well as > restricting searches as appropriate. > > Thanks, > mat > > On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote: > > > I'm building a minimal abstraction around ES to accomplish fine-grained > > security control around searches based on indexed fields. The requirements > > are: > > > > 1. allow users to query ES via search but only return indexed > > documents which they have permission to read > > 2. filter sensitive fields from the results. I'm doing this via > > something like: searchRequestBuilder.addPartialField("apiFields", null, > > FIELDS\_TO\_EXCLUDE); > > > > For the implementation I'm basically composing a SearchRequestBuilder via > > the Java API. > > > > Given a search query in JSON format, what's the best method to set the > > query on the SearchRequestBuilder object while still being able to add my > > filter(s) and exclude fields from the search results? > > > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > > seem to be candidates but it's unclear what exactly they do. Could you > > explain them? > > > > Thanks! -- ---
**Author:** ![Hendrik](https://avatars.discourse-cdn.com/v4/letter/h/839c29/32.png) [@Hendrik](https://discuss.elastic.co/u/Hendrik)\ **Post date:** [November 20, 2013, 9:18am UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/8 "2013-11-20T09:18:23Z")
Maybe this is interesting [https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/tavroa3Nw5g](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/tavroa3Nw5g) Am Freitag, 12. Oktober 2012 22:34:53 UTC+2 schrieb S. Milwerns: > Hi Mat, > > Your approach sounds very interesting. Could you please give me some > example code (e.g. for the "owner" and "specific fields" types)? > > Thank you > Mil > > On Sunday, May 27, 2012 6:20:02 PM UTC+2, mat taylor wrote: > > > Not a direct answer to your question but I am using a simple abstraction > > to implement a fast, very flexible and fine grained security over elastic > > search using a percolater proxy. > > > > The idea is to use a proxy (I am usng nginx + lua) to represent each http > > request as a json document with top level fields such as 'headers', 'body', > > 'path', 'session', and 'method', 'time' and then request authorisation for > > this request by percolating this document through an index of 'policy' > > filters which may or may not match. > > > > For example a policy query could then only be registered > > > > - allow all requests from a specific IP address > > - Use script filters to allow all Update requests only for documents with > > an 'owner' that match user Id supplied as a parameter > > - all access to docs for a limited time > > - only allow access to docs when specific cookies or tokens are available > > in the header or session. > > - only allow queries which request specific fields > > - only updates to specific fields of docs which a user has read access to. > > > > etc.. > > > > If no direct authorization for a request is found, I then the proxy will > > fallback to a set of rewrite rules to add/remove query, field and > > partial\_field parameters to the request to on a per index/type basis. This > > allows sensitive fields to be removed from any search results as well as > > restricting searches as appropriate. > > > > Thanks, > > mat > > > > On Thursday, May 24, 2012 8:56:59 AM UTC-7, Shane Witbeck wrote: > > > > > I'm building a minimal abstraction around ES to accomplish fine-grained > > > security control around searches based on indexed fields. The requirements > > > are: > > > > > > 1. allow users to query ES via search but only return indexed > > > documents which they have permission to read > > > 2. filter sensitive fields from the results. I'm doing this via > > > something like: searchRequestBuilder.addPartialField("apiFields", null, > > > FIELDS\_TO\_EXCLUDE); > > > > > > For the implementation I'm basically composing a SearchRequestBuilder > > > via the Java API. > > > > > > Given a search query in JSON format, what's the best method to set the > > > query on the SearchRequestBuilder object while still being able to add my > > > filter(s) and exclude fields from the search results? > > > > > > SearchRequestBuilder.setSource and SearchRequestBuilder.setExtraSource > > > seem to be candidates but it's unclear what exactly they do. Could you > > > explain them? > > > > > > Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:06am UTC](https://discuss.elastic.co/t/fine-grained-security-for-es-based-api/7844/9 "2017-07-06T02:06:02Z")