Hey all, yet another topic here related to my efforts at reducing ELK's footprint.
This time I'm trying to figure out downsampling. I've successfully configured my dev environment to downsample data. I have downsample* indices with docs in them that I can view via _search.
But I'm still a bit confused on how you actually use the downsampled data.
Specifically I thought I'd be able to find the data while searching in Kibana's Discover view. And that the data would be there for various visualizations and dashboards. After reading the docs, and looking at the json I get via _search on a downsample* index, I'm thinking that that isn't the case, at least for the visualizations and dashboards.
But I'd still expect to be able to find them in Kibana's Discover view, even if I'd have to make a custom Data View pointed at downsample* indices. The thing is, when I make that data view, I get told the @timestamp (nor any other timestamp field) field isn't available, even though I see it in my console search results. And, for some reason, Kibana just doesn't load any of the docs from the index. I guess because it thinks there isn't a timestamp field?
Then, as I was writing this, I figured out Kibana had decided to apply the time range from my metrics-* view to the downsample view. So I reset that to "the last 30 days", and looked at the downsample views again. In the temp view pointed at the full name of a specific downsample index, I was able to see the docs. In the temp view pointed at an actual pattern of down*, I get nothing.
Which leaves me wondering how I'm supposed to use downsampled data. Am I supposed to only look at data from a single downsample index at a time? That just doesn't seem right to me, so I know I'm missing something.
I think I've found most of the documentation on the topic. But after reading through it fairly thoroughly, I haven't connected what's there to the answers to my questions.
- Downsample | Elasticsearch Guide [8.9] | Elastic
- Downsampling a time series data stream | Elasticsearch Guide [8.9] | Elastic
- Downsample index API | Elasticsearch Guide [8.9] | Elastic
Google hasn't been any help either.
So, anyone want to chime in?
How are you supposed to use downsampled data?
I'm currently running ELKStack 8.9.1. Metrics are coming via Elastic Agent. Everything is just running in docker on my laptop.
Thanks in advance!