Nested object facets with filtering by nested object and parent field


(Kaspars Sprogis) #1

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck, hope
somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features, maybe
there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/efdc048c-742d-4f08-8648-704bda0c0af6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(smhdiu) #2

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:
https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features, maybe
there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09c4420e-b15e-40ae-893f-3672e4641765%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kaspars Sprogis) #3

Mohsin, thanks for suggestion, but it doesn't change anything, because i am
using "include_in_parent": true, which means i can query without nested
query too.
Here i updated one of queries with nested query and results are the same:

One more interesting fact i just found out. If i try using "filter facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:
https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03b85529-f9bf-4936-8b30-62551419f739%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Hannes Korte) #4

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because i am
using "include_in_parent": true, which means i can query without nested
query too.
Here i updated one of queries with nested query and results are the same:
https://gist.github.com/darklow/7957463

One more interesting fact i just found out. If i try using "filter facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:
https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52AC6ECF.5030402%40hkorte.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kaspars Sprogis) #5

Thank you Hannes, your query really work, it gives me few new ideas to try.
Actually it's strange, just by removing query by nested object, everything
started working.

Unfortunately now i have another problem, since you moved "genre": "comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because i
am
using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
https://gist.github.com/darklow/7957463

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for
one

movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:

https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to
each

nested object, but this is not the only field from parent i need to
use in

future, so i don't really like this solution to duplicate so many
parent

fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2aec990-5e2a-4a02-a64e-98f1b4eea718%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Hannes Korte) #6

Sounds like you need sub-aggregations. This is a new feature coming in
ES 1.0:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations.html

Unfortunately, I'm not sure whether there already is something
equivalent to your nested facet_filter for aggregations or not. Good luck!

Best regards
Hannes

On 14.12.2013 16:09, Kaspars Sprogis wrote:

Thank you Hannes, your query really work, it gives me few new ideas to try.
Actually it's strange, just by removing query by nested object, everything
started working.

Unfortunately now i have another problem, since you moved "genre": "comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because i
am
using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
https://gist.github.com/darklow/7957463

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for
one

movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:

https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to
each

nested object, but this is not the only field from parent i need to
use in

future, so i don't really like this solution to duplicate so many
parent

fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52AC88FD.3040708%40hkorte.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kaspars Sprogis) #7

Hi,

I just tried new aggregation framework 1.0.0 beta2 and it looks like it is
a bit too early, and such feature is not yet ready.
I also noticed in beta version nested aggregations doesn't work so good
too: https://github.com/elasticsearch/elasticsearch/issues/4449
Anyway thanks, looks like i will have denormalize data or use "filter
facet", until aggregation fw is more complete.

Any new ideas, are still welcome :slight_smile:

On Saturday, December 14, 2013 6:36:13 PM UTC+2, Hannes Korte wrote:

Sounds like you need sub-aggregations. This is a new feature coming in
ES 1.0:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations.html

Unfortunately, I'm not sure whether there already is something
equivalent to your nested facet_filter for aggregations or not. Good luck!

Best regards
Hannes

On 14.12.2013 16:09, Kaspars Sprogis wrote:

Thank you Hannes, your query really work, it gives me few new ideas to
try.
Actually it's strange, just by removing query by nested object,
everything
started working.

Unfortunately now i have another problem, since you moved "genre":
"comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for
different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in
real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types
person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because
i

am

using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
https://gist.github.com/darklow/7957463

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for
one

movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:

https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to
each

nested object, but this is not the only field from parent i need to
use in

future, so i don't really like this solution to duplicate so many
parent

fields in every nested object.

I have been trying to achieve this for few days, and without any
luck,

hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dea6ff5a-ae3b-4d6e-9386-08a23e4959ed%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Kaspars Sprogis) #8

I was wrong - new aggregation totally fixes this problem, with help of
ElasticSearch developer @jpountz, i got my problem fixed.

If anyone was having same problem than me, take a look on query in
following issue, it resolves my problem and gives you a lot more potential
than you could imagine.

On Saturday, December 14, 2013 9:59:53 PM UTC+2, Kaspars Sprogis wrote:

Hi,

I just tried new aggregation framework 1.0.0 beta2 and it looks like it
is a bit too early, and such feature is not yet ready.
I also noticed in beta version nested aggregations doesn't work so good
too: https://github.com/elasticsearch/elasticsearch/issues/4449
Anyway thanks, looks like i will have denormalize data or use "filter
facet", until aggregation fw is more complete.

Any new ideas, are still welcome :slight_smile:

On Saturday, December 14, 2013 6:36:13 PM UTC+2, Hannes Korte wrote:

Sounds like you need sub-aggregations. This is a new feature coming in
ES 1.0:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/search-aggregations.html

Unfortunately, I'm not sure whether there already is something
equivalent to your nested facet_filter for aggregations or not. Good
luck!

Best regards
Hannes

On 14.12.2013 16:09, Kaspars Sprogis wrote:

Thank you Hannes, your query really work, it gives me few new ideas to
try.
Actually it's strange, just by removing query by nested object,
everything
started working.

Unfortunately now i have another problem, since you moved "genre":
"comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for
different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in
real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types
person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything,
because i

am

using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
https://gist.github.com/darklow/7957463

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object
"credits".

What i need is to get facets for departments for specific person
for

one

movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:

https://gist.github.com/darklow/7957463

One of idea was to de-normalize data and add "genre" field also to
each

nested object, but this is not the only field from parent i need to
use in

future, so i don't really like this solution to duplicate so many
parent

fields in every nested object.

I have been trying to achieve this for few days, and without any
luck,

hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation
features,

maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c455b189-17f8-4b96-b4b3-cd8ac7d17f74%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #9