Document Level Permissions Filtering

Elastic Stack Elasticsearch
1

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

2

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or even
index.
And when a user query comes , incercept the query in between using a proxy
or something , see if the user have permission and then only let it through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown richardwhatever@gmail.comwrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

3

Shay,

Building on this question, are you considering adding an internal security
model to ES?

If so, have you looked at what was done in the Apache Accumulo project?
Short story, this was a Big Table implementation built out by the US
Intelligence Community and built out a finer grained security model that
still scaled. If you are considering adding ES security, I'm curious if you
think this level of granularity is desirable/feasible.

http://incubator.apache.org/accumulo/user_manual_1.4-incubating/Security.html

--Mike

On Fri, Mar 23, 2012 at 2:50 PM, Vineeth Mohan vineethmohan@algotree.comwrote:

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or even
index.
And when a user query comes , incercept the query in between using a proxy
or something , see if the user have permission and then only let it through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown richardwhatever@gmail.comwrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

4

Security in search engines is a wide subject.

The "small solution" is just to hand over authorization tokens, index them,
and later add filters to all queries automatically. This could be done by a
web front-end proxy placed before the Elasticsearch index/search REST API.

I think the most preferred way to have document-level security would be a
"big solution", i.e. writing an Elasticsearch security plugin with the
following features included

  • transport layer security between client and server (TLS)
  • server awareness of client identity (establish ACL/role based security)
  • secure indexing: only allow authorized clients to write documents
    (perform ACL/role-based checks)
  • secure query: index visibility, document visibility, filter query results
    due to client authorization
  • enforce document / index data integrity (checksums, end-to-end security)
  • constraint violation alerting
  • and, really advanced stuff, adjust highlighting to filter out sensitive
    (restricted) fields (privacy)
  • even more "paranoid" stuff

Most of the work could be taken over by front-end web processes, and the
Elasticsearch security plugin could offer some conventions for a security
model, i.e. rules for where index/document-level TLS/ACL/Role stuff is
stored and how it is used by clients via a REST API.

Best regards,

Jörg

5

Agreed, security / ACL is a big topic. For document level one, the best way
is to associate the relevant data with each document, and filter by it.

On Sun, Mar 25, 2012 at 1:46 PM, Jörg Prante joergprante@gmail.com wrote:

Security in search engines is a wide subject.

The "small solution" is just to hand over authorization tokens, index
them, and later add filters to all queries automatically. This could be
done by a web front-end proxy placed before the Elasticsearch index/search
REST API.

I think the most preferred way to have document-level security would be a
"big solution", i.e. writing an Elasticsearch security plugin with the
following features included

  • transport layer security between client and server (TLS)
  • server awareness of client identity (establish ACL/role based security)
  • secure indexing: only allow authorized clients to write documents
    (perform ACL/role-based checks)
  • secure query: index visibility, document visibility, filter query
    results due to client authorization
  • enforce document / index data integrity (checksums, end-to-end security)
  • constraint violation alerting
  • and, really advanced stuff, adjust highlighting to filter out sensitive
    (restricted) fields (privacy)
  • even more "paranoid" stuff

Most of the work could be taken over by front-end web processes, and the
Elasticsearch security plugin could offer some conventions for a security
model, i.e. rules for where index/document-level TLS/ACL/Role stuff is
stored and how it is used by clients via a REST API.

Best regards,

Jörg

6

Hi Vineeth,

Sorry for delay in response, i was out of the country.

My requirements are not about filtering the incoming search queries, but
rather about restricting the indexed documents to ones a user has
permission to search... i think it has to be done by the search engine or
paging etc would not work properly.

Regards,

Richard

On 23 March 2012 18:50, Vineeth Mohan vineethmohan@algotree.com wrote:

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or even
index.
And when a user query comes , incercept the query in between using a proxy
or something , see if the user have permission and then only let it through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown richardwhatever@gmail.comwrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/richardbrownni

7

If you add relevant security aspects to the documents indexed, you can
filter by them.

On Thu, Apr 5, 2012 at 9:17 PM, Richard Brown richardwhatever@gmail.comwrote:

Hi Vineeth,

Sorry for delay in response, i was out of the country.

My requirements are not about filtering the incoming search queries, but
rather about restricting the indexed documents to ones a user has
permission to search... i think it has to be done by the search engine or
paging etc would not work properly.

Regards,

Richard

On 23 March 2012 18:50, Vineeth Mohan vineethmohan@algotree.com wrote:

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or even
index.
And when a user query comes , incercept the query in between using a
proxy or something , see if the user have permission and then only let it
through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown <richardwhatever@gmail.com

wrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/richardbrownni

8

Hi Richard & all,

Did you find a solution for this problem ? I have quite the same
requirments/ questions about differents sources (out of ManifoldCF that
seems to manage the whole bunch)....

Thanks.
Yann
Le jeudi 5 avril 2012 20:17:09 UTC+2, Richard Brown a écrit :

Hi Vineeth,

Sorry for delay in response, i was out of the country.

My requirements are not about filtering the incoming search queries, but
rather about restricting the indexed documents to ones a user has
permission to search... i think it has to be done by the search engine or
paging etc would not work properly.

Regards,

Richard

On 23 March 2012 18:50, Vineeth Mohan <vineet...@algotree.com<javascript:>

wrote:

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or even
index.
And when a user query comes , incercept the query in between using a
proxy or something , see if the user have permission and then only let it
through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown <richard...@gmail.com<javascript:>

wrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/richardbrownni

--

9

Hi Yann,

No, I didn't implement a Solr Solution as yet. Currently looking into using
RavenDB instead as its based on lucene.net, but with security built in.

Richard

On 21 November 2012 10:57, Yann Barraud yann.barraud@gmail.com wrote:

Hi Richard & all,

Did you find a solution for this problem ? I have quite the same
requirments/ questions about differents sources (out of ManifoldCF that
seems to manage the whole bunch)....

Thanks.
Yann
Le jeudi 5 avril 2012 20:17:09 UTC+2, Richard Brown a écrit :

Hi Vineeth,

Sorry for delay in response, i was out of the country.

My requirements are not about filtering the incoming search queries, but
rather about restricting the indexed documents to ones a user has
permission to search... i think it has to be done by the search engine or
paging etc would not work properly.

Regards,

Richard

On 23 March 2012 18:50, Vineeth Mohan vineet...@algotree.com wrote:

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or even
index.
And when a user query comes , incercept the query in between using a
proxy or something , see if the user have permission and then only let it
through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown richard...@gmail.comwrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/**richardbrownnihttp://uk.linkedin.com/in/richardbrownni

--

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/richardbrownni

--

10

Hi,

May be not useful idea in your particular use case but in many
organizations document-user visibility is driven by organization
structure/groups, i.e. not really at the level of individual documents. If
that would fit to your situation then you can consider indexing documents
into indices by "organization departments" and then allow users to search
across relevant indices only. Also index aliases could be useful in this
scenario.

Just an idea.

Regards,
Lukáš

--

11

Started this to fulfill our company needs:

Am Sonntag, 25. März 2012 13:46:53 UTC+2 schrieb Jörg Prante:

Security in search engines is a wide subject.

The "small solution" is just to hand over authorization tokens, index
them, and later add filters to all queries automatically. This could be
done by a web front-end proxy placed before the Elasticsearch index/search
REST API.

I think the most preferred way to have document-level security would be a
"big solution", i.e. writing an Elasticsearch security plugin with the
following features included

  • transport layer security between client and server (TLS)
  • server awareness of client identity (establish ACL/role based security)
  • secure indexing: only allow authorized clients to write documents
    (perform ACL/role-based checks)
  • secure query: index visibility, document visibility, filter query
    results due to client authorization
  • enforce document / index data integrity (checksums, end-to-end security)
  • constraint violation alerting
  • and, really advanced stuff, adjust highlighting to filter out sensitive
    (restricted) fields (privacy)
  • even more "paranoid" stuff

Most of the work could be taken over by front-end web processes, and the
Elasticsearch security plugin could offer some conventions for a security
model, i.e. rules for where index/document-level TLS/ACL/Role stuff is
stored and how it is used by clients via a REST API.

Best regards,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

Great effort. Do you plan to add JAAS?

Cheers,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

yes (via tomcat), but i guess the focus will be on kerberos.
My long term plan is to add also things like:

Am Donnerstag, 31. Oktober 2013 14:59:25 UTC+1 schrieb Jörg Prante:

Great effort. Do you plan to add JAAS?

Cheers,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

there is some progress here:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/tavroa3Nw5g

Am Mittwoch, 6. November 2013 12:34:07 UTC+1 schrieb Hendrik:

yes (via tomcat), but i guess the focus will be on kerberos.
My long term plan is to add also things like:

Am Donnerstag, 31. Oktober 2013 14:59:25 UTC+1 schrieb Jörg Prante:

Great effort. Do you plan to add JAAS?

Cheers,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

15

Maybe this is interesting for you

Am Donnerstag, 22. November 2012 18:37:05 UTC+1 schrieb Richard Brown:

Hi Yann,

No, I didn't implement a Solr Solution as yet. Currently looking into
using RavenDB instead as its based on lucene.net, but with security built
in.

Richard

On 21 November 2012 10:57, Yann Barraud <yann.b...@gmail.com <javascript:>

wrote:

Hi Richard & all,

Did you find a solution for this problem ? I have quite the same
requirments/ questions about differents sources (out of ManifoldCF that
seems to manage the whole bunch)....

Thanks.
Yann
Le jeudi 5 avril 2012 20:17:09 UTC+2, Richard Brown a écrit :

Hi Vineeth,

Sorry for delay in response, i was out of the country.

My requirements are not about filtering the incoming search queries, but
rather about restricting the indexed documents to ones a user has
permission to search... i think it has to be done by the search engine or
paging etc would not work properly.

Regards,

Richard

On 23 March 2012 18:50, Vineeth Mohan vineet...@algotree.com wrote:

you will have to implement that on proxy level or something.
Like you need to set a permission level on each document or type or
even index.
And when a user query comes , incercept the query in between using a
proxy or something , see if the user have permission and then only let it
through.

Thanks
Vineeth

On Wed, Mar 21, 2012 at 5:53 PM, Richard Brown richard...@gmail.comwrote:

Hi,

I'm looking for a search engine that provides document level access /
filtering. I had looked at ManifoldCF with Solr as one option. Does
elasticsearch offer this kind of functionality? I need to be able to
restrict search results in a granual way, whereby each document can have
permissions set for each potential user of the system.

Thanks,

Richard.

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/richardbrownni

--

--
Richard Brown
@richardwhatever
uk.linkedin.com/in/richardbrownni

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2747756-f386-4118-a8e3-6489be218a36%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

16

Hi,

Have you implemented this? Can you please give me an idea. I am planning to use kibana for search. If we index user_id or someother data of a document, how do we filter it in user search.

17

You could use a filtered alias and add something like ngnix on top of Elasticsearch.

Or use Shield.

18

Is it possible to give document level permission in shield?

19

Yes. See https://www.elastic.co/guide/en/shield/current/setting-up-field-and-document-level-security.html