Reverse nested aggregation within nested filter aggregation fails

selvinaz_karahancer8 · January 21, 2015, 12:21pm

Hello,

I have to realize distinct data queries on data persisted in ElasticSearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation, but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/50b5bd03-4305-4575-8399-f122d84c4b53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

selvinaz_karahancer8 · January 22, 2015, 10:13am

I am using Elasticsearch 1.3.4.

Nobody has an idea why the buckets of LINES are empty?
Is there another possibility to resolve this problem?

Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
Karahancer-Bouraga:

Hello,

I have to realize distinct data queries on data persisted in Elasticsearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation, but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

masaru · January 23, 2015, 4:22am

Hi,

Not sure if it solves your issue but I think there are a few things to fix:

“attributes" is under “source". nested aggregation’s “path” would be “source.attributes”. You’d need to update field names accordingly as well.
reverse_nested aggregation’s “path” would be empty since it’s joined back to root.
nested aggregation is needed around “LINE” aggregation since you are in root level.

Masaru

On January 22, 2015 at 19:13:10, Selvinaz Karahancer-Bouraga (selvinaz.karahancer84@gmail.com) wrote:

I am using Elasticsearch 1.3.4.

Nobody has an idea why the buckets of LINES are empty?
Is there another possibility to resolve this problem?

Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
Karahancer-Bouraga:

Hello,

I have to realize distinct data queries on data persisted in Elasticsearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation, but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.54c1ccb0.41a7c4c9.1877%40citra.local.
For more options, visit https://groups.google.com/d/optout.

selvinaz_karahancer8 · January 23, 2015, 9:28am

Hi Masaru,

the datamodel is correct, attributes are in the root element, thas was just
a copy paste failure.
But the reason why it fails was that the nested aggregation around “LINE”
has missing.

Now I get the correct results.

Thank you soooo much

2015-01-23 5:22 GMT+01:00 Masaru Hasegawa haniomasali@gmail.com:

Hi,

Not sure if it solves your issue but I think there are a few things to fix:

“attributes" is under “source". nested aggregation’s “path” would be
“source.attributes”. You’d need to update field names accordingly as well.

reverse_nested aggregation’s “path” would be empty since it’s joined
back to root.

nested aggregation is needed around “LINE” aggregation since you are in
root level.

Masaru

On January 22, 2015 at 19:13:10, Selvinaz Karahancer-Bouraga (
selvinaz.karahancer84@gmail.com) wrote:

I am using Elasticsearch 1.3.4.

Nobody has an idea why the buckets of LINES are empty?
Is there another possibility to resolve this problem?

Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
Karahancer-Bouraga:

Hello,

I have to realize distinct data queries on data persisted in
Elasticsearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{

"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" :
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation,
but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch"
group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/XRJNtckzlmE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.54c1ccb0.41a7c4c9.1877%40citra.local
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALEQBoggDfU_LZ_8AetCEJTbWcPHDTDNwVm76HhKt7-v9aRxPg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.