Kibana data values after rollup

Elastic Stack Kibana
1

Hi,
in an index (raw_data) where i have raw data i have records every minute. Each record has always the same value for a field called loss >>> LOSS: 100
raw_data

Then i created a rollup job for raw_data index, to rollup data every 5 minutes. Rollup is running, but when i am checking data in that new rollup index there i see two issues:

  1. data are there not every 5minute but only every 1 hour. Why?
  2. Values are 200 or 300 or 500. How is it possible since in raw_data value is always 100?
    rollup data

This is an example for device A.
When checking different device, records in rollup index are correct, every 5minutes + with correct values.

Thanks

2

Hi,

Can you tell us what version of Kibana you're using and how you created the rollup jobs? Did you use the Kibana interface or the API? If you used Kibana maybe you could post a screenshot of the rollup configuration.

Regards,
Lee

3

Hi,
Kibana version is 7.10.2.
It was via Kibana UI:

image
image562×643 28.6 KB

image
Thanks

4

When you're looking at this data, I think you're look at in in Discover? Do these values change when you select a smaller time range? I'm not sure what version of Kibana this changed in, but you used to see text below the histogram in Discover that showed the duration of each bar. You can manually set that as well but you may be using the Auto time interval.

5

Well, for the first time i noticed it while creating a visualization, then i check the values in Discover.
Values are not changing, they are same. I do not see any reasonable explanation why i am getting values like this.

6

I tried a simple rollup test and I also found problems in Discover. I filed this issue Discover shows wrong values for rollup indices · Issue #136866 · elastic/kibana · GitHub

But what I also found is that if you look at the rollup date with some visualization the data appears correct. As I noted in that issue I filed, I can create a Data Table in Lens and the data appears as I expect. Please see if this also gives you the expected results.

7

I am not sure if we are talking here about same problem. As written in my origin post i see two issues:

  1. data are there not every 5minute but only every 1 hour. Why? // i mean why in roll up index i have data not every 5 minutes but only every hour.
  2. Values are 200 or 300 or 500. How is it possible since in raw_data value is always 100? // how is it possible to get value 200 or 300 in rollup, if the raw data has always value 100.

I also see the wrong data in visualuzations. It was the first place where i noticed it.

Raw data values:
image

Values after rollup:

image
image913×769 18.8 KB

Thanks

8

I'm reaching out to a few other people for suggestions in debugging your issue. You mentioned that;

How many different devices do you have rollup jobs for?
Do any of your rollup jobs have 1 hour intervals?
Are the index names, and the rollup index names all unique enough that they couldn't interfere with each other?
Was this problematic rollup job created before the other working ones, or after? I'm thinking there must be something different either about the input data or the rollup job configuration itself. Have you tried deleting this rollup job and the rollup index and re-creating it?

9

Around 200 devices.
Yes, i have 1h interval too. There it is even more strange, records i have in 5min are missing in 1h and opposite.
5min is called 5min-stats, 1h is called 1h-stats
Yes, i tried to delete the job and create it again, but same results.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.