How are you supposed to use downsampled TSDS data?

Elastic Stack Elasticsearch
1

Hey all, yet another topic here related to my efforts at reducing ELK's footprint.

This time I'm trying to figure out downsampling. I've successfully configured my dev environment to downsample data. I have downsample* indices with docs in them that I can view via _search.

But I'm still a bit confused on how you actually use the downsampled data.

Specifically I thought I'd be able to find the data while searching in Kibana's Discover view. And that the data would be there for various visualizations and dashboards. After reading the docs, and looking at the json I get via _search on a downsample* index, I'm thinking that that isn't the case, at least for the visualizations and dashboards.

But I'd still expect to be able to find them in Kibana's Discover view, even if I'd have to make a custom Data View pointed at downsample* indices. The thing is, when I make that data view, I get told the @timestamp (nor any other timestamp field) field isn't available, even though I see it in my console search results. And, for some reason, Kibana just doesn't load any of the docs from the index. I guess because it thinks there isn't a timestamp field?

Then, as I was writing this, I figured out Kibana had decided to apply the time range from my metrics-* view to the downsample view. So I reset that to "the last 30 days", and looked at the downsample views again. In the temp view pointed at the full name of a specific downsample index, I was able to see the docs. In the temp view pointed at an actual pattern of down*, I get nothing.

Which leaves me wondering how I'm supposed to use downsampled data. Am I supposed to only look at data from a single downsample index at a time? That just doesn't seem right to me, so I know I'm missing something.

I think I've found most of the documentation on the topic. But after reading through it fairly thoroughly, I haven't connected what's there to the answers to my questions.

Google hasn't been any help either.

So, anyone want to chime in?

How are you supposed to use downsampled data?

I'm currently running ELKStack 8.9.1. Metrics are coming via Elastic Agent. Everything is just running in docker on my laptop.

Thanks in advance!

2

I'm going to preface my response here, I have not use the Downsample functionality at all yet, so take what I'm about to say with a bit of caution.

Then, as I was writing this, I figured out Kibana had decided to apply the time range from my metrics-* view to the downsample view. So I reset that to "the last 30 days", and looked at the downsample views again. In the temp view pointed at the full name of a specific downsample index, I was able to see the docs. In the temp view pointed at an actual pattern of down*, I get nothing.

I think there are two possibilities here:

  1. If you're using ILM to do the downsampling, I think that seeing things under metrics-* is correct. This is because ILM will keep the data stream/alias attached to the downsampled index.

  2. If you're doing the downsampling manually, make sure you delete the backing index after you're complete.

A quick side note, I'd check what the downsampled index name and aliases are actually (again I haven't used this feature) and the pattern might not exact match downsample*.

I would agree with you in general though that outside of a bit of docs from the Elasticsearch side regarding downsampling, there isn't a whole lot out there on how to consume it in things like Kibana, making it somewhat hard to try and help without having used the feature before.

3

I have this all set up on two systems, one where I work from home, one in my office. Both are set up to use ILM to manage everything with a super aggressive policy. At home I never was able to tell for sure that it was or wasn't working.

Today I just checked on things here at the office and I think it is working. I was able to filter down to the system.cpu.total.usage.pct (or something like that) field and found results from the downsample index via discover and the metrics-* pattern. Then I checked a metrics dashboard and was able to find the the data there as well.

So I'm chalking this up as an issue with me not looking at things correctly.

Thanks for the reply, I decided to recheck things because of it. :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.