Bitwise operator

George_3 · May 27, 2011, 12:45am

Hi Shay,

First of all thank you for elasticsearch.

Is there a way where I can use bitwise operator on integer field to
filter a search query based on group membership where each group is
assigned to a bit in an integer field.

Say a index with the following type is created:

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/1' -d
'{ "name" : "document A",
"title" : "testing Elastic Search access
control",
"groupmembership": 25}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/2' -d
'{ "name" : "document B",
"title" : "testing Elastic Search access
control",
"groupmembership": 19}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/3' -d
'{ "name" : "document C",
"title" : "testing Elastic Search access
control",
"groupmembership": 11}'

Hence the Group membership of each document would be:
document A = 0000 0000 0001 1001
document B = 0000 0000 0001 0011
document C = 0000 0000 0000 1011

And the following users and their groupmembership are:
"user 01" has the following groupmembership: 0000 ... 0000 0000
0001 (ie int 1)
"user 02" has the following groupmembership: 0000 ... 0000 0000
1000 (ie int 8)
"user 03" has the following groupmembership: 0000 ... 0000 0001
0000 (ie int 16)
"user 04" has the following groupmembership: 0000 ... 0000 0000
0010 (ie int 2)

Is there way to use an integer bitwise filter in elasticsearch where I
can make query that returned
specific documents according to users groupmembership.

example:

query made by using "user 01" groupmembership bitwise filter would

return
document A, document B, and document C

query made by using "user 02" groupmembership bitwise filter would

return
document A, document C

query made by using "user 03" groupmembership bitwise filter would

return
document A, document B

query made by using "user 04" groupmembership bitwise filter would

return
document B, document C

I know that lucene has extension to do this, but I am not sure how I
can do this on elasticsearch.

I know that I could create a proxy wrapper to this. However it would
not be very efficient, as the query itself would have to return all
document matching other filter rules. In addition it would not work
well for query with "size" and "from" restriction. For example I would
not be able to create query which get the first five document for
"user 02" in a single query.

Is there a better way to do this, or should I be using the faceting
facility to solve this issue?

Many Thanks for your help.

kimchy · May 27, 2011, 11:01am

Which extensions does lucene has to do this? I can think of several ways of implementing something like that, but I wonder what you look for.

On Friday, May 27, 2011 at 3:45 AM, George wrote:

Hi Shay,

First of all thank you for elasticsearch.

Is there a way where I can use bitwise operator on integer field to
filter a search query based on group membership where each group is
assigned to a bit in an integer field.

Say a index with the following type is created:

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/1' -d
'{ "name" : "document A",
"title" : "testing Elastic Search access
control",
""groupmembership": 25}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/2' -d
'{ "name" : "document B",
"title" : "testing Elastic Search access
control",
"groupmembership": 19}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/3' -d
'{ "name" : "document C",
"title" : "testing Elastic Search access
control",
"groupmembership": 11}'

Hence the Group membership of each document would be:
document A = 0000 0000 0001 1001
document B = 0000 0000 0001 0011
document C = 0000 0000 0000 1011

And the following users and their groupmembership are:
"user 01" has the following groupmembership: 0000 ... 0000 0000
0001 (ie int 1)
"user 02" has the following groupmembership: 0000 ... 0000 0000
1000 (ie int 8)
"user 03" has the following groupmembership: 0000 ... 0000 0001
0000 (ie int 16)
"user 04" has the following groupmembership: 0000 ... 0000 0000
0010 (ie int 2)

Is there way to use an integer bitwise filter in elasticsearch where I
can make query that returned
specific documents according to users groupmembership.

example:

query made by using "user 01" groupmembership bitwise filter would
return
document A, document B, and document C

query made by using "user 02" groupmembership bitwise filter would
return
document A, document C

query made by using "user 03" groupmembership bitwise filter would
return
document A, document B

query made by using "user 04" groupmembership bitwise filter would
return
document B, document C

I know that lucene has extension to do this, but I am not sure how I
can do this on elasticsearch.

I know that I could create a proxy wrapper to this. However it would
not be very efficient, as the query itself would have to return all
document matching other filter rules. In addition it would not work
well for query with "size" and "from" restriction. For example I would
not be able to create query which get the first five document for
"user 02" in a single query.

Is there a better way to do this, or should I be using the faceting
facility to solve this issue?

Many Thanks for your help.

George_3 · May 27, 2011, 11:28am

Thank you for your reply,

I was thinking of extension made by Israel Ekpo, please see

https://issues.apache.org/jira/browse/LUCENE-2460

I am not too familiar with the architecture of Lucene, as well as the
internal workings of elasticsearch. I guest what I am looking for is a
simple bitwise operator which would allows me to perform query similar
to the following SQL command in elasticsearch.

Assuming we have
currentUserAccessLevelVariable = (int64) 25;

To be able to perform the following psudo sql query:
SELECT column1, column2 ..., AccessRightColumn
FROM someTableName
WHERE bitwise_AND( AccessRightColumn,
currentUserAccessLevelVariable ) > 0

This way I could potentially perform search against 64 attibutes with
single filter comparison operator.

Again, many thanks for your respond.

On May 27, 9:01 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Which extensions does lucene has to do this? I can think of several ways of implementing something like that, but I wonder what you look for.

On Friday, May 27, 2011 at 3:45 AM, George wrote:

Hi Shay,

First of all thank you for elasticsearch.

Is there a way where I can use bitwise operator on integer field to
filter a search query based on group membership where each group is
assigned to a bit in an integer field.

Say a index with the following type is created:

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/1'-d
'{ "name" : "document A",
"title" : "testing Elastic Search access
control",
""groupmembership": 25}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/2'-d
'{ "name" : "document B",
"title" : "testing Elastic Search access
control",
"groupmembership": 19}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/3'-d
'{ "name" : "document C",
"title" : "testing Elastic Search access
control",
"groupmembership": 11}'

Hence the Group membership of each document would be:
document A = 0000 0000 0001 1001
document B = 0000 0000 0001 0011
document C = 0000 0000 0000 1011

And the following users and their groupmembership are:
"user 01" has the following groupmembership: 0000 ... 0000 0000
0001 (ie int 1)
"user 02" has the following groupmembership: 0000 ... 0000 0000
1000 (ie int 8)
"user 03" has the following groupmembership: 0000 ... 0000 0001
0000 (ie int 16)
"user 04" has the following groupmembership: 0000 ... 0000 0000
0010 (ie int 2)

Is there way to use an integer bitwise filter in elasticsearch where I
can make query that returned
specific documents according to users groupmembership.

example:

query made by using "user 01" groupmembership bitwise filter would
return
document A, document B, and document C

query made by using "user 02" groupmembership bitwise filter would
return
document A, document C

query made by using "user 03" groupmembership bitwise filter would
return
document A, document B

query made by using "user 04" groupmembership bitwise filter would
return
document B, document C

I know that lucene has extension to do this, but I am not sure how I
can do this on elasticsearch.

I know that I could create a proxy wrapper to this. However it would
not be very efficient, as the query itself would have to return all
document matching other filter rules. In addition it would not work
well for query with "size" and "from" restriction. For example I would
not be able to create query which get the first five document for
"user 02" in a single query.

Is there a better way to do this, or should I be using the faceting
facility to solve this issue?

Many Thanks for your help.

kimchy · May 27, 2011, 11:32am

I see, well, it uses what I thought, which is the field cache. It can be easily added as a new type of filter in elasticsearch, as elasticsearch exposes the full lucene query stack, and its (quite) simple to plug in new types of filters / queries and have them exposed in the query DSL. Open an issue.

On Friday, May 27, 2011 at 2:28 PM, George wrote:

Thank you for your reply,

I was thinking of extension made by Israel Ekpo, please see

[LUCENE-2460] Search Results Filtering Based on Bitwise Operations on Integer Fields - ASF JIRA

I am not too familiar with the architecture of Lucene, as well as the
internal workings of elasticsearch. I guest what I am looking for is a
simple bitwise operator which would allows me to perform query similar
to the following SQL command in elasticsearch.

Assuming we have
currentUserAccessLevelVariable = (int64) 25;

To be able to perform the following psudo sql query:
SELECT column1, column2 ..., AccessRightColumn
FROM someTableName
WHERE bitwise_AND( AccessRightColumn,
currentUserAccessLevelVariable ) > 0

This way I could potentially perform search against 64 attibutes with
single filter comparison operator.

Again, many thanks for your respond.

On May 27, 9:01 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Which extensions does lucene has to do this? I can think of several ways of implementing something like that, but I wonder what you look for.

On Friday, May 27, 2011 at 3:45 AM, George wrote:

Hi Shay,

First of all thank you for elasticsearch.

Is there a way where I can use bitwise operator on integer field to
filter a search query based on group membership where each group is
assigned to a bit in an integer field.

Say a index with the following type is created:

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/1'-d
'{ "name" : "document A",
"title" : "testing Elastic Search access
control",
""groupmembership": 25}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/2'-d
'{ "name" : "document B",
"title" : "testing Elastic Search access
control",
"groupmembership": 19}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/3'-d
'{ "name" : "document C",
"title" : "testing Elastic Search access
control",
"groupmembership": 11}'

Hence the Group membership of each document would be:
document A = 0000 0000 0001 1001
document B = 0000 0000 0001 0011
document C = 0000 0000 0000 1011

And the following users and their groupmembership are:
"user 01" has the following groupmembership: 0000 ... 0000 0000
0001 (ie int 1)
"user 02" has the following groupmembership: 0000 ... 0000 0000
1000 (ie int 8)
"user 03" has the following groupmembership: 0000 ... 0000 0001
0000 (ie int 16)
"user 04" has the following groupmembership: 0000 ... 0000 0000
0010 (ie int 2)

Is there way to use an integer bitwise filter in elasticsearch where I
can make query that returned
specific documents according to users groupmembership.

example:

query made by using "user 01" groupmembership bitwise filter would
return
document A, document B, and document C

query made by using "user 02" groupmembership bitwise filter would
return
document A, document C

query made by using "user 03" groupmembership bitwise filter would
return
document A, document B

query made by using "user 04" groupmembership bitwise filter would
return
document B, document C

I know that lucene has extension to do this, but I am not sure how I
can do this on elasticsearch.

I know that I could create a proxy wrapper to this. However it would
not be very efficient, as the query itself would have to return all
document matching other filter rules. In addition it would not work
well for query with "size" and "from" restriction. For example I would
not be able to create query which get the first five document for
"user 02" in a single query.

Is there a better way to do this, or should I be using the faceting
facility to solve this issue?

Many Thanks for your help.

George_3 · May 27, 2011, 11:45am

I had opened an issue. Please see the following:

github.com/elastic/elasticsearch

Request to add new Integer Bitwise operator for query DSL

opened 11:44AM - 27 May 11 UTC

closed 04:03PM - 29 Sep 14 UTC

gluhur

>feature discuss feedback_needed :Search/Mapping :Search/Search

Please add bitwise operator which would allows query similar to the following SQ…L command in elasticsearch. Assuming we have currentUserAccessLevelVariable = (int64) 25; To be able to perform the following psudo sql query: SELECT column1, column2 ..., AccessRightColumn FROM someTableName WHERE bitwise_AND( AccessRightColumn, currentUserAccessLevelVariable ) > 0

On May 27, 9:32 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

I see, well, it uses what I thought, which is the field cache. It can be easily added as a new type of filter in elasticsearch, as elasticsearch exposes the full lucene query stack, and its (quite) simple to plug in new types of filters / queries and have them exposed in the query DSL. Open an issue.

On Friday, May 27, 2011 at 2:28 PM, George wrote:

Thank you for your reply,

I was thinking of extension made by Israel Ekpo, please see

[LUCENE-2460] Search Results Filtering Based on Bitwise Operations on Integer Fields - ASF JIRA

I am not too familiar with the architecture of Lucene, as well as the
internal workings of elasticsearch. I guest what I am looking for is a
simple bitwise operator which would allows me to perform query similar
to the following SQL command in elasticsearch.

Assuming we have
currentUserAccessLevelVariable = (int64) 25;

To be able to perform the following psudo sql query:
SELECT column1, column2 ..., AccessRightColumn
FROM someTableName
WHERE bitwise_AND( AccessRightColumn,
currentUserAccessLevelVariable ) > 0

This way I could potentially perform search against 64 attibutes with
single filter comparison operator.

Again, many thanks for your respond.

On May 27, 9:01 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Which extensions does lucene has to do this? I can think of several ways of implementing something like that, but I wonder what you look for.

On Friday, May 27, 2011 at 3:45 AM, George wrote:

Hi Shay,

First of all thank you for elasticsearch.

Is there a way where I can use bitwise operator on integer field to
filter a search query based on group membership where each group is
assigned to a bit in an integer field.

Say a index with the following type is created:

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/1'-d
'{ "name" : "document A",
"title" : "testing Elastic Search access
control",
""groupmembership": 25}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/2'-d
'{ "name" : "document B",
"title" : "testing Elastic Search access
control",
"groupmembership": 19}'

curl -XPUT 'http://localhost:9200/testindex/restricteddocument/3'-d
'{ "name" : "document C",
"title" : "testing Elastic Search access
control",
"groupmembership": 11}'

Hence the Group membership of each document would be:
document A = 0000 0000 0001 1001
document B = 0000 0000 0001 0011
document C = 0000 0000 0000 1011

And the following users and their groupmembership are:
"user 01" has the following groupmembership: 0000 ... 0000 0000
0001 (ie int 1)
"user 02" has the following groupmembership: 0000 ... 0000 0000
1000 (ie int 8)
"user 03" has the following groupmembership: 0000 ... 0000 0001
0000 (ie int 16)
"user 04" has the following groupmembership: 0000 ... 0000 0000
0010 (ie int 2)

Is there way to use an integer bitwise filter in elasticsearch where I
can make query that returned
specific documents according to users groupmembership.

example:

query made by using "user 01" groupmembership bitwise filter would
return
document A, document B, and document C

query made by using "user 02" groupmembership bitwise filter would
return
document A, document C

query made by using "user 03" groupmembership bitwise filter would
return
document A, document B

query made by using "user 04" groupmembership bitwise filter would
return
document B, document C

I know that lucene has extension to do this, but I am not sure how I
can do this on elasticsearch.

I know that I could create a proxy wrapper to this. However it would
not be very efficient, as the query itself would have to return all
document matching other filter rules. In addition it would not work
well for query with "size" and "from" restriction. For example I would
not be able to create query which get the first five document for
"user 02" in a single query.

Is there a better way to do this, or should I be using the faceting
facility to solve this issue?

Many Thanks for your help.