Is there planned work to include the option to drop partial buckets in Kibana Lens visualizations?
We are currently trying to display data aggregations grouped by year, but when we select a specific year, the data from the beginning of the following year are included as well (e.g., if we select 2018, the data from Jan 1st of 2019 at 00:00:00.000 are presented as well). It's quite confusing to viewers of the data then to see a partial bucket of the following year.
I know this option exists in the legacy visualization tools, so am wondering if the similar option will be included in Lens?
This is not currently planned, your best workaround is to match your date filter to the rounded values that you are visualizing. Starting in 7.14 Lens will highlight the partial data ranges like to the legacy visualization tools, which might help with the potential confusion.
@wylie Ah OK, thank you. Our use case here is to show a table of # of downloads per year from our site, for say 2015-2020. Then, the viewer of the data would ideally be able to click on the "2017" filter on the table to show the data for specifically that year. But what happens is the fall-over data from the very beginning of 2018 is being shown too. So unfortunately, given this use case, I don't think it's possible to modify the date filter, since the date filter is generated by the Lens the visualization itself upon clicking '2017'.
If there was an option to drop partial buckets (or more specifically, use an "Exclusive" or "Less than (but not equal to)" filter on the end date), that would accomplish exactly what we need.
I think for now our best workaround is to revert back to the legacy visualization tools, but hopefully this option will be available in future releases.
Can you try setting the time range from January 1st, 2015 at 00:00:00 to December 31st, 2020 at 23:59:59? The issue here is the date histogram rounding behavior, so if you set your time filter to be exactly the start of the interval you won't have issues.
Another option is to use the Filters aggregation instead of the Date Histogram aggregation.
Yes, I'm able to manually set the end time to Dec 31st, 2020 at 23:59:59, which presents the right bucket of data for each year; however, since I'm using the date histogram bucketing in Lens, when I click on the "+" filter for the year in the Lens table, the date range that's generated is from Jan 1st. 2017 - Jan 1st. 2018, which then leads to the very beginning of 2018 to be shown, and I'm not able to manually intervene in that process.
The other option you suggest to use Filters aggregation instead of the Date Histogram does work as [temporary] alternative though, by manually specifying each year's time filter as below:
Obviously this is still less than ideal though, since I'll need to manually do this for every year in our data, and would become unfeasible if we ever wished to show this breakdown by month or by day. So, the option for dropping partial buckets would still be very beneficial in the future.
@wylie, Thank you for creating and following up on the issues! Certainly it would be helpful if there was a way to drop partial buckets in Lens directly, and I think the TimePicker change to allow "less-than" the end date as an option would be even more valuable. I say that because--after thinking about this issue some more--usually we will have more than one Lens visualization (e.g., a pie chart) alongside the Lens date-histogram data table, in which the date range for the pie chart would be controlled by the TimePicker and not the date-histogram.
In other words, even if I was able to create a Lens data table using a date histogram that dropped partial buckets, all of my other Lens visualizations that don't use a date histogram (e.g., pie charts, metrics, etc.), would still be presenting the spill-over data from the very beginning of the End Date value. Consequently, if a user was to click on "2017", the table might say "1,000,000 downloads" for that year, but the pie chart next to the table might say "1,000,005 downloads" due to an extra 5 downloads having occurred at 2018-01-01 00:00:00. Naturally, this could cause some confusion to the viewer.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
