Aggregation bug? Or user error?

mooky · April 29, 2014, 12:04pm

I am seeing some very odd aggregation results - where the sum of the
sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.

"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873",

"0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91FBA",
"33A946DC2B0E494EB371993D345F52E4", "6471AA50DFCF4192B8DD1C2E72A032C7",
"9FB2FFDC0FF0797FE04014AC6F0616B6", "9FB2FFDC0FF1797FE04014AC6F0616B6",
"9FB2FFDC0FF2797FE04014AC6F0616B6", "9FB2FFDC0FF3797FE04014AC6F0616B6",
"9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6",
"AFE0FED33F06AFB6E04015AC5E060AA3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
"trafficOperatorSid" : "S-1-5-21-20xxxxxx"
}
}
},
"NOT_ME" : {
"filter" : {
"not" : {
"filter" : {
"term" : {
"trafficOperatorSid" : "S-1-5-21-20xxxxxx"
}
}
}
}
}
}
}
}
},

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edc64188-0d1a-423f-9cc8-ec13842ec970%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mooky · April 29, 2014, 5:35pm

It looks like a bug to me - but if its user error, then obviously I can fix
it a lot quicker

On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:

I am seeing some very odd aggregation results - where the sum of the
sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.
"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873", 
"0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91FBA",
"33A946DC2B0E494EB371993D345F52E4", "6471AA50DFCF4192B8DD1C2E72A032C7",
"9FB2FFDC0FF0797FE04014AC6F0616B6", "9FB2FFDC0FF1797FE04014AC6F0616B6",
"9FB2FFDC0FF2797FE04014AC6F0616B6", "9FB2FFDC0FF3797FE04014AC6F0616B6",
"9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6",
"AFE0FED33F06AFB6E04015AC5E060AA3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
"trafficOperatorSid" : "S-1-5-21-20xxxxxx"<span
style="color: #000;" class="styled-by
...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jpountz · April 30, 2014, 1:13pm

This looks wrong indeed. By any chance, would you have a curl recreation of
this issue?

On Tue, Apr 29, 2014 at 7:35 PM, mooky nick.minutello@gmail.com wrote:

It looks like a bug to me - but if its user error, then obviously I can
fix it a lot quicker

On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:
I am seeing some very odd aggregation results - where the sum of the
sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.
"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873", "
0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91FBA", "
33A946DC2B0E494EB371993D345F52E4", "6471AA50DFCF4192B8DD1C2E72A032C7", "
9FB2FFDC0FF0797FE04014AC6F0616B6", "9FB2FFDC0FF1797FE04014AC6F0616B6", "
9FB2FFDC0FF2797FE04014AC6F0616B6", "9FB2FFDC0FF3797FE04014AC6F0616B6", "
9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6", "
AFE0FED33F06AFB6E04015AC5E060AA3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
              "trafficOperatorSid" : "S-1-5-21-20xxxxxx"<span
style="color: #000;" class="styled-by
...
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7WWj4GaAEH0K%2B37srpP4f_9S%3DKffM7k1DAAyZiy1zUpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

mooky · May 2, 2014, 10:07am

I havent been able to figure out what is required to recreate it.
I am doing a number of identical aggregations (just different values intentMarketCode
and intentDate
Three aggregations give correct numbers - one doesnt.... I havent figured
why....

On Wednesday, 30 April 2014 14:13:00 UTC+1, Adrien Grand wrote:

This looks wrong indeed. By any chance, would you have a curl recreation
of this issue?

On Tue, Apr 29, 2014 at 7:35 PM, mooky <nick.mi...@gmail.com <javascript:>

wrote:
It looks like a bug to me - but if its user error, then obviously I can
fix it a lot quicker

On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:
I am seeing some very odd aggregation results - where the sum of the
sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.
"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873", "
0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91FBA", "
33A946DC2B0E494EB371993D345F52E4", "6471AA50DFCF4192B8DD1C2E72A032C7", "
9FB2FFDC0FF0797FE04014AC6F0616B6", "9FB2FFDC0FF1797FE04014AC6F0616B6", "
9FB2FFDC0FF2797FE04014AC6F0616B6", "9FB2FFDC0FF3797FE04014AC6F0616B6", "
9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6", "
AFE0FED33F06AFB6E04015AC5E060AA3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
              "trafficOperatorSid" : "S-1-5-21-20xxxxxx"<span 
style="color: #000;" class="styled-by
...
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jpountz · May 2, 2014, 1:34pm

What version of Elasticsearch are you using? If it is small enough, I would
also be interested if you could share your index so that I can try to
reproduce the issue locally.

On Fri, May 2, 2014 at 12:07 PM, mooky nick.minutello@gmail.com wrote:

I havent been able to figure out what is required to recreate it.
I am doing a number of identical aggregations (just different values intentMarketCode
and intentDate
Three aggregations give correct numbers - one doesnt.... I havent figured
why....

On Wednesday, 30 April 2014 14:13:00 UTC+1, Adrien Grand wrote:
This looks wrong indeed. By any chance, would you have a curl recreation
of this issue?

On Tue, Apr 29, 2014 at 7:35 PM, mooky nick.mi...@gmail.com wrote:
It looks like a bug to me - but if its user error, then obviously I can
fix it a lot quicker

On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:
I am seeing some very odd aggregation results - where the sum of the
sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.
"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873",
"0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91FBA",
"33A946DC2B0E494EB371993D345F52E4", "6471AA50DFCF4192B8DD1C2E72A032C7",
"9FB2FFDC0FF0797FE04014AC6F0616B6", "9FB2FFDC0FF1797FE04014AC6F0616B6",
"9FB2FFDC0FF2797FE04014AC6F0616B6", "9FB2FFDC0FF3797FE04014AC6F0616B6",
"9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6",
"AFE0FED33F06AFB6E04015AC5E060AA3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
              "trafficOperatorSid" : "S-1-5-21-20xxxxxx"<span
style="color: #000;" class="styled-by
...
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%
40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7kdz9N%3DOi6mtjWwO480FigztUcAnjouvmyt7Y0vu00Tw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

mooky · May 6, 2014, 9:34am

I am using elastic 1.1.1.
The index isn't huge (600m) - but it contains financially sensitive data...
will be too problematic legally to allow it offsite. I can try anonymise
the data - see if it can be reproduced that way - might learn something
about what is causing it.

On Friday, 2 May 2014 14:34:21 UTC+1, Adrien Grand wrote:

What version of Elasticsearch are you using? If it is small enough, I
would also be interested if you could share your index so that I can try to
reproduce the issue locally.

On Fri, May 2, 2014 at 12:07 PM, mooky <nick.mi...@gmail.com <javascript:>

wrote:
I havent been able to figure out what is required to recreate it.
I am doing a number of identical aggregations (just different values intentMarketCode
and intentDate
Three aggregations give correct numbers - one doesnt.... I havent figured
why....

On Wednesday, 30 April 2014 14:13:00 UTC+1, Adrien Grand wrote:
This looks wrong indeed. By any chance, would you have a curl recreation
of this issue?

On Tue, Apr 29, 2014 at 7:35 PM, mooky nick.mi...@gmail.com wrote:
It looks like a bug to me - but if its user error, then obviously I can
fix it a lot quicker

On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:
I am seeing some very odd aggregation results - where the sum of the
sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.
"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873", 
"0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91FBA"
, "33A946DC2B0E494EB371993D345F52E4", "6471AA50DFCF4192B8DD1C2E72A032
C7", "9FB2FFDC0FF0797FE04014AC6F0616B6", "
9FB2FFDC0FF1797FE04014AC6F0616B6", "9FB2FFDC0FF2797FE04014AC6F0616B6",
"9FB2FFDC0FF3797FE04014AC6F0616B6", "9FB2FFDC0FF5797FE04014AC6F0616B6"
, "9FB2FFDC0FF6797FE04014AC6F0616B6", "AFE0FED33F06AFB6E04015AC5E060A
A3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
              "trafficOperatorSid" : "S-1-5-21-20xxxxxx"<span 
style="color: #000;" class="styled-by
...
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%
40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f3bcf03c-cfe3-40f9-b599-45f9a8d4c973%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mooky · June 3, 2014, 4:37pm

I have managed to produce a unit test that exposes this (albeit different
to the data above).
The index is quite small - and the data fictional - so theres no problem
sending you the index.

Here is a result I get - and we can see the sub-aggregations have higher
counts than the parent:
{
"sales_quotas": {
"doc_count": 6,
"shipmentDate": {
"buckets": [
{
"key": "Overdue",
"to": 1.3989024E12,
"to_as_string": "2014-05-01",
"doc_count": 2,
"nothingAllocated": {
"doc_count": 6,
"ME": {
"doc_count": 0
},
"NOT_ME": {
"doc_count": 6
}
}
}
]
}
}
}

On Tuesday, 6 May 2014 10:34:53 UTC+1, mooky wrote:

I am using elastic 1.1.1.
The index isn't huge (600m) - but it contains financially sensitive
data... will be too problematic legally to allow it offsite. I can try
anonymise the data - see if it can be reproduced that way - might learn
something about what is causing it.

On Friday, 2 May 2014 14:34:21 UTC+1, Adrien Grand wrote:
What version of Elasticsearch are you using? If it is small enough, I
would also be interested if you could share your index so that I can try to
reproduce the issue locally.

On Fri, May 2, 2014 at 12:07 PM, mooky nick.mi...@gmail.com wrote:
I havent been able to figure out what is required to recreate it.
I am doing a number of identical aggregations (just different values intentMarketCode
and intentDate
Three aggregations give correct numbers - one doesnt.... I havent
figured why....

On Wednesday, 30 April 2014 14:13:00 UTC+1, Adrien Grand wrote:
This looks wrong indeed. By any chance, would you have a curl
recreation of this issue?

On Tue, Apr 29, 2014 at 7:35 PM, mooky nick.mi...@gmail.com wrote:
It looks like a bug to me - but if its user error, then obviously I
can fix it a lot quicker

On Tuesday, 29 April 2014 13:04:53 UTC+1, mooky wrote:
I am seeing some very odd aggregation results - where the sum of
the sub-aggregations is more than the parent bucket.

Results:
"CSSX" : {
"doc_count" : 24,
"intentDate" : {
"buckets" : [ {
"key" : "Overdue",
"to" : 1.3981248E12,
"to_as_string" : "2014-04-22",
"doc_count" : 1,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "May",
"from" : 1.3981248E12,
"from_as_string" : "2014-04-22",
"to" : 1.4006304E12,
"to_as_string" : "2014-05-21",
"doc_count" : 23,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
}, {
"key" : "June",
"from" : 1.4006304E12,
"from_as_string" : "2014-05-21",
"to" : 1.4033088E12,
"to_as_string" : "2014-06-21",
"doc_count" : 0,
"ME" : {
"doc_count" : 0
},
"NOT_ME" : {
"doc_count" : 24
}
} ]
}
},

I wouldn't have thought that to be possible at all.
Here is the request that generated the dodgy results.
"CSSX" : {
  "filter" : {
    "and" : {
      "filters" : [ {
        "type" : {
          "value" : "inventory"
        }
      }, {
        "term" : {
          "isAllocated" : false
        }
      }, {
        "term" : {
          "intentMarketCode" : "CSSX"
        }
      }, {
        "terms" : {
          "groupCompanyId" : [ "0D13EF2D0E114D43BFE362F5024D8873"
, "0D593DE0CFBE49BEA3BF5AD7CD965782", "1E9C36CC45C64FCAACDEE0AF4FB91F
BA", "33A946DC2B0E494EB371993D345F52E4", "
6471AA50DFCF4192B8DD1C2E72A032C7", "9FB2FFDC0FF0797FE04014AC6F0616B6"
, "9FB2FFDC0FF1797FE04014AC6F0616B6", "9FB2FFDC0FF2797FE04014AC6F0616
B6", "9FB2FFDC0FF3797FE04014AC6F0616B6", "
9FB2FFDC0FF5797FE04014AC6F0616B6", "9FB2FFDC0FF6797FE04014AC6F0616B6"
, "AFE0FED33F06AFB6E04015AC5E060AA3" ]
}
}, {
"not" : {
"filter" : {
"terms" : {
"status" : [ "Cancelled", "Completed" ]
}
}
}
} ]
}
},
"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-04-22"
}, {
"key" : "May",
"from" : "2014-04-22",
"to" : "2014-05-21"
}, {
"key" : "June",
"from" : "2014-05-21",
"to" : "2014-06-21"
} ]
},
"aggregations" : {
"ME" : {
"filter" : {
"term" : {
              "trafficOperatorSid" : "S-1-5-21-20xxxxxx"<span 
style="color: #000;" class="styled-by
...
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4ceceaaf-4fb8-4e54-97f4-c49fcbf9493d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3e7d8928-f76b-4358-97b9-3189e037006c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.
--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dda7db79-c31f-48c0-b137-955ba054a1b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mooky · June 3, 2014, 4:41pm

By the way this test fails with elastic 1.2 also.

How do I go about uploading an index with aggregation request json, etc?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jpountz · June 3, 2014, 8:40pm

A recreation would be really great! If you can zip it and upload it to any
file sharing service, that would work for me.

On Tue, Jun 3, 2014 at 6:41 PM, mooky nick.minutello@gmail.com wrote:

By the way this test fails with elastic 1.2 also.

How do I go about uploading an index with aggregation request json, etc?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j68foACRpstuJ-nGTgANgR%3Dqk%2Bh%3DTaSw2mRgRQiauY9%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

mooky · June 4, 2014, 11:33am

Bah.
I thought I had a simple unit test that was reliably recreating it - but it
would appear not. Its still very intermittent - and my test never seems to
fail when run on its own.

....

On Tuesday, 3 June 2014 21:41:04 UTC+1, Adrien Grand wrote:

A recreation would be really great! If you can zip it and upload it to any
file sharing service, that would work for me.

On Tue, Jun 3, 2014 at 6:41 PM, mooky <nick.mi...@gmail.com <javascript:>>
wrote:

By the way this test fails with elastic 1.2 also.

How do I go about uploading an index with aggregation request json, etc?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2284bf7f-5561-40d6-a430-08b4dbbaca00%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1286f2ed-fbce-4145-834e-a579bcf84cb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mooky · June 6, 2014, 6:55pm

Ok. I have written a test case that (if run enough) will reproduce it. Its
an intermittent bug.
I have raised an issue:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0b666a91-2f49-4787-ba2f-fb33a8fc023e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.