Reverse nested aggregation within nested filter aggregation fails

Hello,

I have to realize distinct data queries on data persisted in ElasticSearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation, but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/50b5bd03-4305-4575-8399-f122d84c4b53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I am using Elasticsearch 1.3.4.

Nobody has an idea why the buckets of LINES are empty?
Is there another possibility to resolve this problem?

Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
Karahancer-Bouraga:

Hello,

I have to realize distinct data queries on data persisted in ElasticSearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation, but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

Not sure if it solves your issue but I think there are a few things to fix:

  • “attributes" is under “source". nested aggregation’s “path” would be “source.attributes”. You’d need to update field names accordingly as well.
  • reverse_nested aggregation’s “path” would be empty since it’s joined back to root.
  • nested aggregation is needed around “LINE” aggregation since you are in root level.

Masaru

On January 22, 2015 at 19:13:10, Selvinaz Karahancer-Bouraga (selvinaz.karahancer84@gmail.com) wrote:

I am using Elasticsearch 1.3.4.

Nobody has an idea why the buckets of LINES are empty?
Is there another possibility to resolve this problem?

Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
Karahancer-Bouraga:

Hello,

I have to realize distinct data queries on data persisted in ElasticSearch.
My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{
"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation, but
I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.54c1ccb0.41a7c4c9.1877%40citra.local.
For more options, visit https://groups.google.com/d/optout.

Hi Masaru,

the datamodel is correct, attributes are in the root element, thas was just
a copy paste failure.
But the reason why it fails was that the nested aggregation around “LINE”
has missing.

Now I get the correct results.

Thank you soooo much :slight_smile:

2015-01-23 5:22 GMT+01:00 Masaru Hasegawa haniomasali@gmail.com:

Hi,

Not sure if it solves your issue but I think there are a few things to fix:

  • “attributes" is under “source". nested aggregation’s “path” would be
    “source.attributes”. You’d need to update field names accordingly as well.
  • reverse_nested aggregation’s “path” would be empty since it’s joined
    back to root.
  • nested aggregation is needed around “LINE” aggregation since you are in
    root level.

Masaru

On January 22, 2015 at 19:13:10, Selvinaz Karahancer-Bouraga (
selvinaz.karahancer84@gmail.com) wrote:

I am using Elasticsearch 1.3.4.

Nobody has an idea why the buckets of LINES are empty?
Is there another possibility to resolve this problem?

Am Mittwoch, 21. Januar 2015 13:21:04 UTC+1 schrieb Selvinaz
Karahancer-Bouraga:

Hello,

I have to realize distinct data queries on data persisted in
ElasticSearch.

My data model looks like:

{"took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"event_index_v_0_3","_type":"EventBean","_id":"o6tFCVGjS7mnyUV92d7tOQ","_score":1.0,"_source":{

"severityLevel" : "SL_EVENT",
"source" : {
"sourceId" : "1",
"sourceType" : "VEHICLE",
"description" : null,
"mandator" : {
"mandatorId" : "DEF",
"mandatorName" : null,
"priority" : null
},
"eventTime" : 1410768722000,
"version" : "Version_0_1",
"attributes" : [ {
"paramKey" : "COURSE",
"value" : "123"
}, {
"paramKey" : "DRIVERNO",
"value" : "111"
}, {
"paramKey" : "LINE",
"value" : "101"
}, {
"paramKey" : "gps_x",
"value" : "11111111"
}, {
"paramKey" : "gps_y",
"value" : "87654321"
} ]
}}
where attributes are nested objects of EventBeans.
Now I want to have all distinct values of mandatorId, LINE and gps_x.
The aggregationbuilder looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"terms" : {
"field" : "source.mandator.mandatorId",
"size" : 2147483647,
"min_doc_count" : 1
},
"aggregations" : {
"attributes" : {
"nested" : {
"path" : "attributes"
},
"aggregations" : {
"gps_x" : {
"filter" : {
"term" : {
"attributes.paramKey" : "gps_x"
}
},
"aggregations" : {
"gps_x" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
},
"aggregations" : {
"attributes" : {
"reverse_nested" : {
"path" : "attributes"
},
"aggregations" : {
"LINE" : {
"filter" : {
"term" : {
"attributes.paramKey" : "LINE"
}
},
"aggregations" : {
"LINE" : {
"terms" : {
"field" : "attributes.value",
"size" : 2147483647,
"order" : {
"_count" : "desc"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}

and the response looks like:

"aggregations" : {
"source.mandator.mandatorId" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "def",
"doc_count" : 3,
"attributes" : {
"doc_count" : 15,
"gps_x" : {
"doc_count" : 3,
"gps_x" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "11111111",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "22222222",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
}, {
"key" : "33333333",
"doc_count" : 1,
"attributes" : {
"doc_count" : 1,
"LINE" : {
"doc_count" : 0,
"LINE" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
} ]
}
}
}
} ]
}
}

The buckets of LINEs are empty, I am using reverse_nested aggregation,
but

I think I am still in the filter of attributes.paramKey=gps_x.
How can I solve this problem?
Anyone an idea?

Thank you for your help.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch"
group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/04894ca8-28c8-4ca5-add9-4d4f79802ac9%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/XRJNtckzlmE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.54c1ccb0.41a7c4c9.1877%40citra.local
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALEQBoggDfU_LZ_8AetCEJTbWcPHDTDNwVm76HhKt7-v9aRxPg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.