DateRange aggregation semantics - include_lower/include_upper?

Elastic Stack Elasticsearch
1

What are the expected semantics of the from/to fields in a DateRange
aggregation?

Are the from/to values included? Should there be an include_lower/
include_upper option like with filters?

I want the aggregation to include the lower and upper values - but I
discovered today that it doesnt (I was getting a mismatch between the
aggregation & the filter with gte/lte values).

My aggregation was defined like this:

"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-02-17"
}, {
"key" : "March",
"from" : "2014-02-18",
"to" : "2014-03-17"
}, {
"key" : "April",
"from" : "2014-03-18",
"to" : "2014-04-15T00:00:00.001Z"
} ]
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f38e3ce-2ff9-43e9-a476-c94f82256451%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Oops. I copy-pasted the aggregation I was hacking about with.
The actual aggregation looks like this (I had added an additional
millisecond as a hack to include_upper)

"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-02-17"
}, {
"key" : "March",
"from" : "2014-02-18",
"to" : "2014-03-17"
}, {
"key" : "April",
"from" : "2014-03-18",
"to" : "2014-04-15"
} ]
}
}
}

On Monday, 24 February 2014 18:16:07 UTC, mooky wrote:

What are the expected semantics of the from/to fields in a DateRange
aggregation?

Are the from/to values included? Should there be an include_lower/
include_upper option like with filters?

I want the aggregation to include the lower and upper values - but I
discovered today that it doesnt (I was getting a mismatch between the
aggregation & the filter with gte/lte values).

My aggregation was defined like this:

"aggregations" : {
"intentDate" : {
"date_range" : {
"field" : "intentDate",
"ranges" : [ {
"key" : "Overdue",
"to" : "2014-02-17"
}, {
"key" : "March",
"from" : "2014-02-18",
"to" : "2014-03-17"
}, {
"key" : "April",
"from" : "2014-03-18",
"to" : "2014-04-15T00:00:00.001Z"
} ]
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cd6a4ad2-12ee-4137-aa35-d313fc6a2872%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Yes, you are correct. The "from" is inclusive, and the "to" is exclusive.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/01116905-849f-44a4-aa61-17ec6149ba00%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I think its necessary to be able to specify an include_lower/
include_upper option like with filters.

On Tuesday, 25 February 2014 14:54:24 UTC, Binh Ly wrote:

Yes, you are correct. The "from" is inclusive, and the "to" is exclusive.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/29315df6-645c-40a5-8920-b28488efa6fa%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Does anyone think this is a worthwhile change?

On Wednesday, 26 February 2014 12:41:53 UTC, mooky wrote:

I think its necessary to be able to specify an include_lower/
include_upper option like with filters.

On Tuesday, 25 February 2014 14:54:24 UTC, Binh Ly wrote:

Yes, you are correct. The "from" is inclusive, and the "to" is exclusive.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

In filters we prefer using gt/gte/lt/lte because there you can be specific
about including or excluding, but for aggregations you want "to" to exclude
otherwise you end up with overlapping ranges.

What use case do you have where you want to change that?

clint

On 5 March 2014 14:26, mooky nick.minutello@gmail.com wrote:

Does anyone think this is a worthwhile change?

On Wednesday, 26 February 2014 12:41:53 UTC, mooky wrote:

I think its necessary to be able to specify an include_lower/
include_upper option like with filters.

On Tuesday, 25 February 2014 14:54:24 UTC, Binh Ly wrote:

Yes, you are correct. The "from" is inclusive, and the "to" is exclusive.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKT4k8Oett%3DzjHwzco0KKsoNMrUzgXzR0WSSnkpuK%2BxVQA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hi Clinton
I am doing a date-range aggregation - the ranges are defined by (financial)
exchange rules - and they are just dates (LocalDate's) (ie there is no time
component). The end of the period is what is defined by the exchange
rules.
I am showing a number of things that are "due" in the "current" period, the
previous period and the next period. (But in fact, previous period actually
means all past periods - to catch anything that is overdue).
So the previous, current and next period-end dates are:
2014-02-19, 2014-03-19 & 2014-04-16.
I define the ranges therefore as:
Overdue: date < 2014-02-19
March: 2014-02-20 < date < 2014-03-19
April: 2014-03-20 < date < 2014-04-16

In the User Interface the user sees the number of things that fall between
these periods - and when the click on one of them, the range is translated
to a filter: So, if the user clicks on March, they would expect to see 2014-02-20
< date < 2014-03-19 as the filter criteria.

So the ranges are already mutually exclusive (albeit by defining the start
date as previous-period + 1 day).
That range, though, is what the user would expect. We could define our
range using Period-end-date + 1 day - and then translate that to a filter
criteria by subtracting one day.... but I'd prefer to be able to take the
range definition and make a filter without any voodoo-ing with the dates.
Does that make sense?

I understand what you are saying - in the majority case you probably want
to define (numerical - or even datetime) ranges without having to worry
about avoiding overlap & double-counting. But in my case, the ranges are
already defined so there is no overlap - and the semantics are that "to" is
inclusive.

-Nick

On Wednesday, 5 March 2014 13:28:20 UTC, Clinton Gormley wrote:

In filters we prefer using gt/gte/lt/lte because there you can be specific
about including or excluding, but for aggregations you want "to" to exclude
otherwise you end up with overlapping ranges.

What use case do you have where you want to change that?

clint

On 5 March 2014 14:26, mooky <nick.mi...@gmail.com <javascript:>> wrote:

Does anyone think this is a worthwhile change?

On Wednesday, 26 February 2014 12:41:53 UTC, mooky wrote:

I think its necessary to be able to specify an include_lower/
include_upper option like with filters.

On Tuesday, 25 February 2014 14:54:24 UTC, Binh Ly wrote:

Yes, you are correct. The "from" is inclusive, and the "to" is
exclusive.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0728b685-9692-4da2-858a-d8de0a2ca10c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

Yes? No?

-Nick

On Friday, 7 March 2014 11:46:48 UTC, mooky wrote:

Hi Clinton
I am doing a date-range aggregation - the ranges are defined by
(financial) exchange rules - and they are just dates (LocalDate's) (ie
there is no time component). The end of the period is what is defined
by the exchange rules.
I am showing a number of things that are "due" in the "current" period,
the previous period and the next period. (But in fact, previous period
actually means all past periods - to catch anything that is overdue).
So the previous, current and next period-end dates are:
2014-02-19, 2014-03-19 & 2014-04-16.
I define the ranges therefore as:
Overdue: date < 2014-02-19
March: 2014-02-20 < date < 2014-03-19
April: 2014-03-20 < date < 2014-04-16

In the User Interface the user sees the number of things that fall between
these periods - and when the click on one of them, the range is translated
to a filter: So, if the user clicks on March, they would expect to see 2014-02-20
< date < 2014-03-19 as the filter criteria.

So the ranges are already mutually exclusive (albeit by defining the start
date as previous-period + 1 day).
That range, though, is what the user would expect. We could define our
range using Period-end-date + 1 day - and then translate that to a filter
criteria by subtracting one day.... but I'd prefer to be able to take the
range definition and make a filter without any voodoo-ing with the dates.
Does that make sense?

I understand what you are saying - in the majority case you probably want
to define (numerical - or even datetime) ranges without having to worry
about avoiding overlap & double-counting. But in my case, the ranges are
already defined so there is no overlap - and the semantics are that "to" is
inclusive.

-Nick

On Wednesday, 5 March 2014 13:28:20 UTC, Clinton Gormley wrote:

In filters we prefer using gt/gte/lt/lte because there you can be
specific about including or excluding, but for aggregations you want "to"
to exclude otherwise you end up with overlapping ranges.

What use case do you have where you want to change that?

clint

On 5 March 2014 14:26, mooky nick.mi...@gmail.com wrote:

Does anyone think this is a worthwhile change?

On Wednesday, 26 February 2014 12:41:53 UTC, mooky wrote:

I think its necessary to be able to specify an include_lower/
include_upper option like with filters.

On Tuesday, 25 February 2014 14:54:24 UTC, Binh Ly wrote:

Yes, you are correct. The "from" is inclusive, and the "to" is
exclusive.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2160569c-643c-4ee1-9845-22339b78943f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2bc4ede4-636c-4d1a-8ee7-0f2caecb6652%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

9

On 7 March 2014 12:46, mooky nick.minutello@gmail.com wrote:

So the previous, current and next period-end dates are:
2014-02-19, 2014-03-19 & 2014-04-16.
I define the ranges therefore as:
Overdue: date < 2014-02-19
March: 2014-02-20 < date < 2014-03-19
April: 2014-03-20 < date < 2014-04-16

Actually, it should be:

Overdue: date <= 2014-02-19
March: 2014-02-20 < date <= 2014-03-19
April: 2014-03-20 < date <= 2014-04-16

which would be represented as:

{ to: "2014-02-20"}
{ from: "2014-02-20", to: "2014-03-20" }
{ from: "2014-03-20", to: "2014-04-17" }

I realise that this requires a little work on your side (ie adding a day),
but I'm not pushing back on this just because "it can already be done". I
asked for the same thing a while back. Turns out that the problem is that
the implementation with to/from is very optimized. If we were to support
to/from/incl/excl/gt/gte/lt/lte etc, then the implementation would need to
pass that information around while building the aggs, impacting performance.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQMcOzH%2By-eDKdP-794u-gbDE_HhpZg6O_T%3DmoYpCg01g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

10

Actually, yes, it should be

Overdue: date <= 2014-02-19
March: 2014-02-20 <= date <= 2014-03-19
April: 2014-03-20 <= date <= 2014-04-16

Yeah, I can add a day to the upper-range to work around this (its what I
have to do now).
However, it also means I have to then subtract a day when taking the
results to create a filter definition - because the expected filter
behaviour is gte/lte.

ie If the user clicks on "March", we have to translate that into a
query/filter to show the list of things that fall within that range.
The user sees the filter details (and can modify them) - and the behaviour
the user expects is gte/lte - ie the user will expect to see the date range
2014-02-20--2014-03-19.
So it means I have to do a bunch of flipping back and forth...

I appreciate the performance implications. Is it practical to have an
optimised/flexible implementations?

-Nick

On Tuesday, 11 March 2014 15:19:24 UTC, Clinton Gormley wrote:

On 7 March 2014 12:46, mooky <nick.mi...@gmail.com <javascript:>> wrote:

So the previous, current and next period-end dates are:
2014-02-19, 2014-03-19 & 2014-04-16.
I define the ranges therefore as:
Overdue: date < 2014-02-19
March: 2014-02-20 < date < 2014-03-19
April: 2014-03-20 < date < 2014-04-16

Actually, it should be:

Overdue: date <= 2014-02-19
March: 2014-02-20 < date <= 2014-03-19
April: 2014-03-20 < date <= 2014-04-16

which would be represented as:

{ to: "2014-02-20"}
{ from: "2014-02-20", to: "2014-03-20" }
{ from: "2014-03-20", to: "2014-04-17" }

I realise that this requires a little work on your side (ie adding a day),
but I'm not pushing back on this just because "it can already be done". I
asked for the same thing a while back. Turns out that the problem is that
the implementation with to/from is very optimized. If we were to support
to/from/incl/excl/gt/gte/lt/lte etc, then the implementation would need to
pass that information around while building the aggs, impacting performance.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c6813f07-9c56-4e29-822c-5e0f7b164f32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

11

Is it practical to have an optimised/flexible implementations?

Re-reading, that sentence doesn't make sense.
What I meant to ask is whether it would be practical to have 2
implementations - an optimised one and a flexible one?

-Nick

On Wednesday, 12 March 2014 11:09:44 UTC, mooky wrote:

Actually, yes, it should be

Overdue: date <= 2014-02-19
March: 2014-02-20 <= date <= 2014-03-19
April: 2014-03-20 <= date <= 2014-04-16

Yeah, I can add a day to the upper-range to work around this (its what I
have to do now).
However, it also means I have to then subtract a day when taking the
results to create a filter definition - because the expected filter
behaviour is gte/lte.

ie If the user clicks on "March", we have to translate that into a
query/filter to show the list of things that fall within that range.
The user sees the filter details (and can modify them) - and the behaviour
the user expects is gte/lte - ie the user will expect to see the date range
2014-02-20--2014-03-19.
So it means I have to do a bunch of flipping back and forth...

I appreciate the performance implications. Is it practical to have an
optimised/flexible implementations?

-Nick

On Tuesday, 11 March 2014 15:19:24 UTC, Clinton Gormley wrote:

On 7 March 2014 12:46, mooky nick.mi...@gmail.com wrote:

So the previous, current and next period-end dates are:
2014-02-19, 2014-03-19 & 2014-04-16.
I define the ranges therefore as:
Overdue: date < 2014-02-19
March: 2014-02-20 < date < 2014-03-19
April: 2014-03-20 < date < 2014-04-16

Actually, it should be:

Overdue: date <= 2014-02-19
March: 2014-02-20 < date <= 2014-03-19
April: 2014-03-20 < date <= 2014-04-16

which would be represented as:

{ to: "2014-02-20"}
{ from: "2014-02-20", to: "2014-03-20" }
{ from: "2014-03-20", to: "2014-04-17" }

I realise that this requires a little work on your side (ie adding a
day), but I'm not pushing back on this just because "it can already be
done". I asked for the same thing a while back. Turns out that the
problem is that the implementation with to/from is very optimized. If we
were to support to/from/incl/excl/gt/gte/lt/lte etc, then the
implementation would need to pass that information around while building
the aggs, impacting performance.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0a9b0605-cb1b-4b6a-93d3-777fffd0f209%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.