Nested object facets with filtering by nested object and parent field

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck, hope
somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features, maybe
there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/efdc048c-742d-4f08-8648-704bda0c0af6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features, maybe
there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09c4420e-b15e-40ae-893f-3672e4641765%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Mohsin, thanks for suggestion, but it doesn't change anything, because i am
using "include_in_parent": true, which means i can query without nested
query too.
Here i updated one of queries with nested query and results are the same:

One more interesting fact i just found out. If i try using "filter facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03b85529-f9bf-4936-8b30-62551419f739%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because i am
using "include_in_parent": true, which means i can query without nested
query too.
Here i updated one of queries with nested query and results are the same:
Nested facets filtering by parent in ElasticSearch · GitHub

One more interesting fact i just found out. If i try using "filter facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to each
nested object, but this is not the only field from parent i need to use in
future, so i don't really like this solution to duplicate so many parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52AC6ECF.5030402%40hkorte.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thank you Hannes, your query really work, it gives me few new ideas to try.
Actually it's strange, just by removing query by nested object, everything
started working.

Unfortunately now i have another problem, since you moved "genre": "comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because i
am
using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
Nested facets filtering by parent in ElasticSearch · GitHub

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for
one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to
each
nested object, but this is not the only field from parent i need to
use in
future, so i don't really like this solution to duplicate so many
parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2aec990-5e2a-4a02-a64e-98f1b4eea718%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sounds like you need sub-aggregations. This is a new feature coming in
ES 1.0:

Unfortunately, I'm not sure whether there already is something
equivalent to your nested facet_filter for aggregations or not. Good luck!

Best regards
Hannes

On 14.12.2013 16:09, Kaspars Sprogis wrote:

Thank you Hannes, your query really work, it gives me few new ideas to try.
Actually it's strange, just by removing query by nested object, everything
started working.

Unfortunately now i have another problem, since you moved "genre": "comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because i
am
using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
Nested facets filtering by parent in ElasticSearch · GitHub

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for
one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to
each
nested object, but this is not the only field from parent i need to
use in
future, so i don't really like this solution to duplicate so many
parent
fields in every nested object.

I have been trying to achieve this for few days, and without any luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52AC88FD.3040708%40hkorte.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

I just tried new aggregation framework 1.0.0 beta2 and it looks like it is
a bit too early, and such feature is not yet ready.
I also noticed in beta version nested aggregations doesn't work so good
too: Aggregation framework - filtering aggregations by nested object fields doesn't work correctly · Issue #4449 · elastic/elasticsearch · GitHub
Anyway thanks, looks like i will have denormalize data or use "filter
facet", until aggregation fw is more complete.

Any new ideas, are still welcome :slight_smile:

On Saturday, December 14, 2013 6:36:13 PM UTC+2, Hannes Korte wrote:

Sounds like you need sub-aggregations. This is a new feature coming in
ES 1.0:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Unfortunately, I'm not sure whether there already is something
equivalent to your nested facet_filter for aggregations or not. Good luck!

Best regards
Hannes

On 14.12.2013 16:09, Kaspars Sprogis wrote:

Thank you Hannes, your query really work, it gives me few new ideas to
try.
Actually it's strange, just by removing query by nested object,
everything
started working.

Unfortunately now i have another problem, since you moved "genre":
"comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for
different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in
real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types
person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything, because
i
am
using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
Nested facets filtering by parent in ElasticSearch · GitHub

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object "credits".

What i need is to get facets for departments for specific person for
one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to
each
nested object, but this is not the only field from parent i need to
use in
future, so i don't really like this solution to duplicate so many
parent
fields in every nested object.

I have been trying to achieve this for few days, and without any
luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dea6ff5a-ae3b-4d6e-9386-08a23e4959ed%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I was wrong - new aggregation totally fixes this problem, with help of
Elasticsearch developer @jpountz, i got my problem fixed.

If anyone was having same problem than me, take a look on query in
following issue, it resolves my problem and gives you a lot more potential
than you could imagine.

On Saturday, December 14, 2013 9:59:53 PM UTC+2, Kaspars Sprogis wrote:

Hi,

I just tried new aggregation framework 1.0.0 beta2 and it looks like it
is a bit too early, and such feature is not yet ready.
I also noticed in beta version nested aggregations doesn't work so good
too: Aggregation framework - filtering aggregations by nested object fields doesn't work correctly · Issue #4449 · elastic/elasticsearch · GitHub
Anyway thanks, looks like i will have denormalize data or use "filter
facet", until aggregation fw is more complete.

Any new ideas, are still welcome :slight_smile:

On Saturday, December 14, 2013 6:36:13 PM UTC+2, Hannes Korte wrote:

Sounds like you need sub-aggregations. This is a new feature coming in
ES 1.0:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Unfortunately, I'm not sure whether there already is something
equivalent to your nested facet_filter for aggregations or not. Good
luck!

Best regards
Hannes

On 14.12.2013 16:09, Kaspars Sprogis wrote:

Thank you Hannes, your query really work, it gives me few new ideas to
try.
Actually it's strange, just by removing query by nested object,
everything
started working.

Unfortunately now i have another problem, since you moved "genre":
"comedy"
out of facet, this means i can have only one facet per genre per query,
which doesn't work for me, since i need more than one facet for
different
genres - that is why i was adding "genre" to facet_filter.

Actually "genre" is just a showcase parameter to show my problem - in
real
world i am using "_type", i just made it "genre" so it is more
understandable.
In real world want to see facets of departments for specific types
person
is involved in.
Which means unfortunately this query doesn't work for me :frowning:
Any ideas how to allow multiple facets per different genres?

On Saturday, December 14, 2013 4:44:31 PM UTC+2, Hannes Korte wrote:

Hi Kaspars,

I think this search request does the trick:

{
"query": {
"term": {
"genre": "comedy"
}
},
"facets": {
"comedy_departments": {
"terms": {
"field": "credits.department"
},
"facet_filter": {
"nested": {
"path": "credits",
"query": {
"match": {
"credits.person_id": "1"
}
},
"join" : false
}
},
"nested" : "credits"
}
},
"size": 0
}

Best regards
Hannes

On 14.12.2013 15:16, Kaspars Sprogis wrote:

Mohsin, thanks for suggestion, but it doesn't change anything,
because i
am
using "include_in_parent": true, which means i can query without
nested
query too.
Here i updated one of queries with nested query and results are the
same:
Nested facets filtering by parent in ElasticSearch · GitHub

One more interesting fact i just found out. If i try using "filter
facet"
count instead of terms and "facet_filter", then count is correct, but
counting for each department separately would be wrong. Strange thing
terms
facets doesn't return same results/counts.

I also updated gist example with facet filter and also added actual
results.

On Saturday, December 14, 2013 3:52:09 PM UTC+2, Mohsin Husen wrote:

nested object can be only queried using nested query

use nested query instead of term query :slight_smile:

Elasticsearch Platform — Find real-time answers at scale | Elastic

HTH
Mohsin

On Saturday, 14 December 2013 10:23:40 UTC, Kaspars Sprogis wrote:

Hi,

I have movies index and each movie contains nested object
"credits".

What i need is to get facets for departments for specific person
for
one
movie genre.
In other words: facets for credit.department where genre=comedy and
credits.person_id=1

Here i setup full example, with mapping and queries i tried with
comments:
Nested facets filtering by parent in ElasticSearch · GitHub

One of idea was to de-normalize data and add "genre" field also to
each
nested object, but this is not the only field from parent i need to
use in
future, so i don't really like this solution to duplicate so many
parent
fields in every nested object.

I have been trying to achieve this for few days, and without any
luck,
hope somebody have some better ideas.
Have anyone tried latest beta version with new aggregation
features,
maybe there are features that can solve my problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c455b189-17f8-4b96-b4b3-cd8ac7d17f74%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.