Aggregations on multiple children cause wrong doc_count

Elastic Stack Elasticsearch
1

I have an index which consists of Groups, Members and Discussions. Both
Discussions and Members are children of Groups.
When doing an aggregation query to find out how many Members and
Discussions there're for a specific Group(s) the aggregation which is 2nd
ends up having a duplicated doc_count.

My query looks like this, membershipCount will end up being double of the
correct value:

{
"query" : {
"ids" : {
"type" : "group",
"values" : [ "group_id" ]
}
},
"aggregations" : {
"groups" : {
"terms" : {
"field" : "id"
},
"aggregations" : {
"discussionCount" : {
"children" : {
"type" : "discussion"
}
},
"membershipCount" : {
"children" : {
"type" : "groupMembership"
}
}
}
}
}
}

If I simply switch membershipCount and discussionCount, making
discussionCount the 2nd subaggregation the doc count for memberships is
correct but discussionCount ends up doubled.

I'm able to get rid of this behavior by running a cardinality
subaggregation like this:

{
"query" : {
"ids" : {
"type" : "group",
"values" : [ "group_id" ]
}
},
"aggregations" : {
"groups" : {
"terms" : {
"field" : "id"
},
"aggregations" : {
"discussionCount" : {
"children" : {
"type" : "discussion"
},
"aggregations" : {
"activeDiscussionAggs" : {
"cardinality" : {
"field" : "id"
}
}
}
},
"membershipCount" : {
"children" : {
"type" : "groupMembership"
},
"aggregations" : {
"groupMembers" : {
"cardinality" : {
"field" : "id"
}
}
}
}
}
}
}
}

Adding "collect_mode" : "breadth_first" to the terms query even causes the
first doc_count to end up being 3times the correct count and the 2nd
aggregation count 6times the correct count.
Using cardinality everything works fine but I'd greatly appreciate if
anyone could explain this behavior or if my aggregation should be done a
different way.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/759c356f-94f2-43eb-b3f0-6a6d3a1089bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hard to diagnose the query behaviour without the docs + mappings.
Can you post examples?

On Friday, February 20, 2015 at 10:26:19 PM UTC, matt...@netbrains.com
wrote:

I have an index which consists of Groups, Members and Discussions. Both
Discussions and Members are children of Groups.
When doing an aggregation query to find out how many Members and
Discussions there're for a specific Group(s) the aggregation which is 2nd
ends up having a duplicated doc_count.

My query looks like this, membershipCount will end up being double of the
correct value:

{
"query" : {
"ids" : {
"type" : "group",
"values" : [ "group_id" ]
}
},
"aggregations" : {
"groups" : {
"terms" : {
"field" : "id"
},
"aggregations" : {
"discussionCount" : {
"children" : {
"type" : "discussion"
}
},
"membershipCount" : {
"children" : {
"type" : "groupMembership"
}
}
}
}
}
}

If I simply switch membershipCount and discussionCount, making
discussionCount the 2nd subaggregation the doc count for memberships is
correct but discussionCount ends up doubled.

I'm able to get rid of this behavior by running a cardinality
subaggregation like this:

{
"query" : {
"ids" : {
"type" : "group",
"values" : [ "group_id" ]
}
},
"aggregations" : {
"groups" : {
"terms" : {
"field" : "id"
},
"aggregations" : {
"discussionCount" : {
"children" : {
"type" : "discussion"
},
"aggregations" : {
"activeDiscussionAggs" : {
"cardinality" : {
"field" : "id"
}
}
}
},
"membershipCount" : {
"children" : {
"type" : "groupMembership"
},
"aggregations" : {
"groupMembers" : {
"cardinality" : {
"field" : "id"
}
}
}
}
}
}
}
}

Adding "collect_mode" : "breadth_first" to the terms query even causes the
first doc_count to end up being 3times the correct count and the 2nd
aggregation count 6times the correct count.
Using cardinality everything works fine but I'd greatly appreciate if
anyone could explain this behavior or if my aggregation should be done a
different way.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2894615c-752b-41f0-9d61-e48ef2a10322%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Here're the mappings:

{
"discussion" : {
"mappings" : {
"groupMembership" : {
"_parent" : {
"type" : "group"
},
"_routing" : {
"required" : true
},
"properties" : {
"approved" : {
"type" : "boolean",
"store" : true
},
"id" : {
"type" : "string",
"index" : "not_analyzed"
},
"parent" : {
"type" : "string",
"index" : "not_analyzed",
"store" : true
},
"userId" : {
"type" : "string",
"index" : "not_analyzed"
}
}
},
"group" : {
"properties" : {
"createdDate" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"deletedDate" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"description" : {
"type" : "string"
},
"id" : {
"type" : "string",
"index" : "not_analyzed"
},
"modifiedDate" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"name" : {
"type" : "string",
"store" : true
},
"targetType" : {
"type" : "string"
},
"tenantId" : {
"type" : "string",
"index" : "not_analyzed"
},
"visibility" : {
"type" : "string",
"index" : "not_analyzed"
}
}
},
"discussion" : {
"_parent" : {
"type" : "group"
},
"_routing" : {
"required" : true
},
"properties" : {
"authorId" : {
"type" : "string",
"index" : "not_analyzed"
},
"createdDate" : {
"type" : "date",
"format" : "date_optional_time"
},
"deletedDate" : {
"type" : "date",
"format" : "date_optional_time"
},
"id" : {
"type" : "string",
"index" : "not_analyzed"
},
"message" : {
"type" : "string"
},
"modifiedDate" : {
"type" : "date",
"format" : "date_optional_time"
},
"parent" : {
"type" : "string",
"index" : "not_analyzed"
},
"subject" : {
"type" : "string",
"fields" : {
"subject.raw" : {
"type" : "string",
"index" : "not_analyzed"
}
}
},
"tenantId" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}
}

And some sample objects:
{
"_index" : "discussion",
"_type" : "group",
"_id" : "7cc810d9-1412-48d0-97bd-830543230624",
"_score" : 1.0,
"_source":{"id":"7cc810d9-1412-48d0-97bd-830543230624","name":"CHAPTER 19. The Prophet.","description":""Shipmates, have ye shipped in that ship?"","visibility":"PUBLIC","targetType":null,"createdDate":1424460745000,"modifiedDate":1424460745000,"deletedDate":null,"tenantId":"18b37386-dd53-496a-82b4-26ef56d4fdf9"}
}, {
"_index" : "discussion",
"_type" : "groupMembership",
"_id" : "f8beda9a-87cd-4783-b369-645aa1fecc8a",
"_score" : 1.0,
"_source":{"id":"f8beda9a-87cd-4783-b369-645aa1fecc8a","userId":"54570d06-44ae-4241-9118-f74ea5345703","parent":"7cc810d9-1412-48d0-97bd-830543230624","approved":true}
}, {
"_index" : "discussion",
"_type" : "groupMembership",
"_id" : "98894b52-c764-4b4b-b737-dc9393a5a6b1",
"_score" : 1.0,
"_source":{"id":"98894b52-c764-4b4b-b737-dc9393a5a6b1","userId":"5c04f5bb-8d75-47bb-87a0-095ea3e05818","parent":"7cc810d9-1412-48d0-97bd-830543230624","approved":true}
}, {
"_index" : "discussion",
"_type" : "groupMembership",
"_id" : "f51b93d8-55d9-4a3c-aa4a-e7c2abcd3bbe",
"_score" : 1.0,
"_source":{"id":"f51b93d8-55d9-4a3c-aa4a-e7c2abcd3bbe","userId":"11cdfd7c-59a4-49a8-b3d9-62c4b49bf761","parent":"7cc810d9-1412-48d0-97bd-830543230624","approved":true}
}, {
"_index" : "discussion",
"_type" : "groupMembership",
"_id" : "145b9ae9-daf4-4de7-8245-74453565744b",
"_score" : 1.0,
"_source":{"id":"145b9ae9-daf4-4de7-8245-74453565744b","userId":"c5c103a3-c7c5-41ec-b1c8-0e4aa6861344","parent":"7cc810d9-1412-48d0-97bd-830543230624","approved":true}
}, {
"_index" : "discussion",
"_type" : "discussion",
"_id" : "af2d431d-b8bc-49cc-87cd-6bbe724ad4b8",
"_score" : 1.0,
"_source":{"id":"af2d431d-b8bc-49cc-87cd-6bbe724ad4b8","authorId":"11cdfd7c-59a4-49a8-b3d9-62c4b49bf761","parent":"7cc810d9-1412-48d0-97bd-830543230624","subject":""Look ye; when Captain Ahab is all right, then this left arm of mine will be all right; not before."","message":""What do you know about him?"","createdDate":1424460748000,"modifiedDate":1424460748000,"deletedDate":null,"tenantId":"18b37386-dd53-496a-82b4-26ef56d4fdf9"}
}, {
"_index" : "discussion",
"_type" : "discussion",
"_id" : "cb9ff3f1-9f27-4567-9625-a15cbfdd9544",
"_score" : 1.0,
"_source":{"id":"cb9ff3f1-9f27-4567-9625-a15cbfdd9544","authorId":"5c04f5bb-8d75-47bb-87a0-095ea3e05818","parent":"7cc810d9-1412-48d0-97bd-830543230624","subject":"said I.","message":""HE'S got enough, though, to make up for all deficiencies of that sort in other chaps," abruptly said the stranger, placing a nervous emphasis upon the word HE.","createdDate":1424460747000,"modifiedDate":1424460747000,"deletedDate":null,"tenantId":"18b37386-dd53-496a-82b4-26ef56d4fdf9"}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/858b2645-510e-4e40-8429-4e0fe7a87a2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.