Hi,
We've recently migrated from ES 5.2.1 to ES 6.7.0. However, after migration, we've observed that aggregations involving parent-child relationships are too slow (almost 10 times slower) when compared to our earlier '5.2' cluster.
The parent-child types in '5.2' have now been replaced with 'joins' in the new version. Please see the sample mapping and the aggregation request below to get a gist of our situation:
Sample Mapping:
{
"com.example.mymapping": {
"properties": {
"parentBaseLevel": {
"properties": {
"parentBaseLevelField1": {
"type": "keyword"
},
"parentNestedLevel": {
"type": "nested",
"properties": {
"parentNestedLevelField1": {
"type": "keyword"
},
"parentNestedLevelField12": {
"type": "keyword"
}
}
}
}
},
"childBaseLevel": {
"properties": {
"childBaseLevelField1": {
"type": "boolean"
}
}
},
"my_join_field": {
"type": "join",
"relations": {
"parentBaseLevel": [
"childBaseLevel"
]
}
}
}
}
}
In our previous mapping, we had childBaseLevel and parentBaseLevel as two seperate types within the same index where one is parent of the other.
Aggregation call on the above mapping:
{
"aggregations": {
"nested_parentNestedLevel": {
"aggregations": {
"parentNestedLevelField1": {
"aggregations": {
"reverse_nested_TO_parentNestedLevel": {
"aggregations": {
"child_childBaseLevel": {
"aggregations": {
"filter_childBaseLevelField1": {
"filter": {
"bool": {
"adjust_pure_negative": true,
"boost": 1.0,
"must": [
{
"bool": {
"adjust_pure_negative": true,
"boost": 1.0,
"must": [
{
"term": {
"childBaseLevel.childBaseLevelField1": {
"boost": 1.0,
"value": true
}
}
}
]
}
}
]
}
}
}
},
"children": {
"type": "childBaseLevel"
}
}
},
"reverse_nested": {
}
}
},
"terms": {
"field": "parentBaseLevel.parentNestedLevel.parentNestedLevelField1",
"min_doc_count": 1,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
],
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"size": 2000
}
}
},
"nested": {
"path": "parentBaseLevel.parentNestedLevel"
}
}
}
}
The calls that we make to ES are pretty much exactly the same to that of above. In the above request, if we simply remove the child aggregation, it's really fast. Please note that, we've used the same query in our previous 5.2 cluster and the results were fast.
After going through elastic search reference, we've tried to use ["eager_global_ordinals" : true] for individual specific fields where we are doing terms aggregation ( parentNestedLevelField1 in example above ) and also to the join field ( my_join_field in example above ) but saw no performance improvement. Our use case is more towards search speed than indexing.
The document count is also not much (by scale) the parentType documents count is 2.5 million and the child documents count is 300 K. Overall count of all documents in the index is 3.1 million. We have 3 shards with no replicas.
We'd like to know what has changed from 5.2 to 6.7 that increased the search time for these types of aggregations. Are there any ways in which we can improve our search performance without needing to de-normalize the data?
Any help in this regard is highly appreciated.
Thanks